Understanding dynamic sound in Flash
I made a little test app in order to help me understand how dynamic sound in Flash works. I had done a few of the basic tutorials/experiments and I kind of got the idea, but quite a lot of mystery still remained in the topic... While doing this experiment though, I feel like I have a much more thorough understanding of the way dynamic sound works in Flash and so, I decided to share it, 'cause I would have loved to find something like this on someone elses blog. Besides, I feel like the more people are familiar with this the better because this feature of FP10 is simple awesome!
So, how does dynamic sound work? The new method in Sound called "extract" can read audio data from any position of a Sound object. It extracts the data into a ByteArray, that is, into a bunch of 0s and 1s. But the hardest thing for me to understand was: What is the nature of this data? How does it describe a sound?
Well, byte arrays are highly efficient arrays that contain boolean values, or bits. These 0s and 1s are packed in groups of 8, each being a byte. So byteArray.position = 256 would not place the array pointer at bit 256, it would place it at byte 256, that is, at bit 2048. Now, the extract method populates the byte array with a bunch of bytes, but in order to make sense out of the bytes, we must visualize them as groups of 8 bytes (64 bits). Grouped in this way, each group is what we call a sample in audio...
Real life sound is nothing more than oscillations or vibrations that travel through air. The amplitude of these oscillations correspond to volume and the cycle or wavelength of these oscillations correspond to what we interpret as pitch. The way digital sound works is to "sample" the instantaneous amplitude of these vibrations at a very fast rate (say 44100Hz, that is 44100 samples each second). Such amplitude readings can be stored in 32 bit "float" values. Flash works at 44100Hz, stereo, so the first 4 bytes of each sample correspond to an instantaneous value in the left channel and the next 4 to the amplitude of the right channel. 32 bits per channel. So if we use the extract method, reposition the byte array in 0, we can use the readFloat method of ByteArray in pairs, and hence extract these amplitude values of the entire sound, one by one. Doing this, its crucial to keep in mind that each time you read a part of the byte array, the position marker is shifted to the end of what you've just read. Its just the way byte arrays work, differing from regular arrays.
In this manner, we can read large chunks of samples and give them to the data object in the SampleDataEvent. Doing so, we are delivering a chunk of audio to the computer's audio card so it can make some noise. The beauty in this is that we can read and understand the sound data, and before giving it to the sound card, we can process the audio with complex filters and DSP analysis, use the information for visual display, etc... This is why it is handled in chunks of audio, so we can be able to process this sound in real time.
Its important to note that when we extract a chunk from the Sound object, we don't know if (imagine that we are arriving to the end of a sound) we are going to be able to get the amount of samples we asked for, so the extract method returns a value, indicating how many samples it was able to extract.
So enough talking... see it yourself in the demo by clicking on the image above (view source enabled). In this experiment, I am not doing anything to the original sounds, just using the info extracted each cycle to visualize whats going on. The boxes plot the amplitude of the samples againts time, and from the pixelated look of them, you can easily grasp how many samples these mp3's have... A lot of them. The upper box shows the waveform of the entire sound clip (note that it is not a frequency spectrum), and the lower box shows the instantaneous waveform of the chunk that has just been extracted from the Sound object and been delivered to the sound card. You can change the playback speed or drag the playback head to see the process in slow motion. You can also change the buffer size, which is nothing more that the size of the chunks (in samples) that are processed at a time on each cycle. And, you can change the sounds too, just because waveforms are beautiful... be patient on the loading though, my server is sloooooow...
Processing sounds can be quite cpu intensive, but luckily we have pixel bender and alchemy to give us a little more juice in this area. I'd like to post more about this topic in the future, its just too interesting. Perhaps a spectrum analyzer, pitch shifting, time scale modification, etc... The possibilities are endless. Just look at what guys like Andre Michelle are doing!
Extruding TextField3D
We've added a simple class that provides extrusions for TextField3D objects. Right click on the demo to see the source and learn how to use the feature.
UPDATE: Properties "textColor" and "textAlpha" have been replaced by "material". Since TextField3D supports various material types now, these two properties are not applicable any more.
Away3D currently supports all the basic color materials on irregular shapes like these. You can use WireColorMaterial, ColorMaterial, WireframeMaterial and ShadingColorMaterial on vector text and vector graphics coming from Flash.
There is still a lot of work to do, but we're getting there!
Importing graphics from Flash to Away3D
In a very similar way we used the wumedia package to extract fonts from Flash, we can use it to extract vector graphics in general from Flash. This was possible before in Away3D using VectorSwf.as, as shown on a previous post, but the work flow that I will present now is so much simpler, that we've decided to deprecate VectorSwf.as and replace it with the much more powerful and easier to use Swf.as
Click on the image to see what I'm talking about (right-click view source enabled). The snowman was designed in Flash, and a swf was exported containing this graphic and other vector assets that may want to be used in 3D. After that, the assets swf is loaded using the following code, which is the same way many other external assets are loaded with Away3D, such as Collada, 3ds, Obj, etc:
_loader = Swf.load("assets/vectorassets.swf", {libraryClips:["Snowman"], scaling:2});
"libraryClips" is an array containing all the linkage names of the clips that want to be extracted. It is important to set these linkages in Flash in order to be able to identify the clips from the main app. The parameter is optional however, if not supplied, Swf.as will attempt to extract the graphics from the stage. For this to be possible however, the instances on the stage must be set as graphics types.
Another thing to take into consideration is the flattening of the clips. As one of our current goals is to be able to extrude these clips, Away3D treats flattened graphics differently from graphics that are grouped, or on layers, and non flattened graphics might produce sorting problems at this stage. In order to correctly flatten a vector graphic in Flash (as for instance a drawing imported from Illustrator), the way I recommend it is:
- Alternate "break apart" with "distribute to layers" until no grouped elements are left.
- Select all the resulting shapes and go to Modify -> Shape -> Convert Lines to Fills.
- Copy the content of all the produced layers and paste it into a new clip.
This simple process has ensured me so far to be able to migrate any graphic from Flash to 3D properly. Even pretty complex ones copy-pasted from Illustrator.
Sometimes its good to keep layers however. Imagine a window component that you wish to take to 3D. The window could have a title, a title bar, a frame and a background. And now imagine that you want this window to remain in layers as opposed to being flat (eg, the title bar above the background, title above the title bar). What you do is skip the flattening and use:
{libraryClips:["Snowman"], scaling:2, perspectiveOffset:-100, perspectiveFocus:2000}
in the init object passed to the constructor. This example would separate the layers by 100 pixels backwards, and if viewed exactly 2000 pixels from the front of the container in which the graphics were imported to by Swf.as, you will see exactly what you saw in 2D! With perspective correction of course. As each element is drawn back, it is scaled up so that the overall image doesn't loose proportion in each of its elements. If you move from this focus point however, you should become aware of the layering and the offsets between the graphics.
I hope someone finds usefulness in all this. And please, do provide feedback if you've got any ideas or requests!
TextField3D Tutorial
So, what's new with dynamic 3D text in Away3D now? Not much from the end-user point of view, but the internals have changed dramatically from previous implementations. What's most important is that TextField3D is now a permanent class in the engine. We no longer have to use the textfields branch in order to use vector features. The branch was experimental, is now deprecated, and all the investigated features are now being assimilated in the main trunk.
Dynamic textfields are now using Guojian Wu's powerful swf parser. You can see this magnificent tool and more of his work at wu-media.com. The parser allows us to use any font installed in the system, and provides many text wrapping features such as leading, kerning, alignment, etc.
There are many ways to use it. The basic principle is that Wu's parser identifyes a font's byte data in an swf and extracts its vector information. Font byte data can be in the same swf as the main application, or in a different asset swf. The way I recommend to use it is as follows:
1) Create a new FLA file with Flash, create a dynamic textfield for each font on stage, and on each use the properties panel to embed the characters you will want to be using. Export the swf as, for example, fonts.swf.
2) In another actionscript project, use an URLLoader to load the fonts.swf as binary data.
3) Once the data is loaded, use "VectorText.extractFont(_loader.data)" so that the parser can process the fonts. NOTE: In flex, you can also embed the fonts.swf file, and pass the bytes to extractFont() as "new FontBytes()".
4) You can now create TextField3D instances as:
var textfield:TextField3D = new TextField3D("Arial", {textAlpha:1, textColor:0x000000, text:"TextField3D", size:200, leading:0, kerning:0, textWidth:2000, align:"TL"});
_scene.addChild(textfield);
UPDATE: Properties "textColor" and "textAlpha" have been replaced by "material". Since TextField3D supports various material types now, these two properties are not applicable any more.
Click on the image above to see a demo, and then right-click on the demo to view the source.
What's next? In the next few days we will continue to assimilate features from the textfields branch into Away3D's main trunk. Amongst these coming features are vector text extrusion, support for advanced shading materials on vector primitives like text, importing of sprites from Flash with wu's parser in a very similar way as using text in this tutorial, tessellation tools for vector primitives, extrusion of sprites imported from Flash, etc.
I'll make sure to post any updates on this progress. On the mean time, I hope you enjoy these basic text features, and don't hesitate to post any suggestions here, or in the Away3D user group!
Meta
Categories
Archives
- May 2010
- January 2010
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- April 2009
- February 2009
- January 2009
- December 2008
- October 2008
- September 2008
Links
- Away3D
- EzeQL
- Fabrice Closier
- Infinite Turtles
- Kevin Luck
- London Flash Platform User Group
- TheTechLabs
- wu-media lab
