Supporting Double-Byte character

#1 Samme,

Are you sure you need to make bitmaps for *65536* characters?

Why not only make bitmaps for characters you will be actually *using* in your particular language?

#2 I would suggest implementing a software renderer for the font and storing the result into a texture, then using that texture for as long as the text needs to be displayed.

If you will have constantly changing text, perhaps you can cache data into texture memory in some way...

Of course, if you're storing these characters in a 1bit color depth bitmap, you're only talking about half a meg of texture data.

(Oh yeah - for unicode support, 65k isn't an unreasonable number of characters to want to have... :))

#3 James,

Actually I am not using 65536 bitmaps. Since we are using double byte encoding, and using it directly for indexing the array, I simply make a array of 65536 size.

As we are doing online game, we cannot limit the Chinese character user can use. In such case we need to group all possible characters for display. I am using about 32767 characters, covering 0x8000 to 0xffff double byte.

I also considering to rewrite or implement a brand new system for Asia language support.

#4

Quote:Will it cause too much performance drop?

It depens on wich videocard will run that. I think the best solution is develop a new system. But if you dont have the time go with that

#5 I am working on how to display Chinese, anyone above can help me?

#6 Like Ben said, you can look into storing that texture as a 1-bit bitmap, it'll be very small, even if it's huge.

But, I'm not sure if today's videocards support 1-bit textures. If not, you'll have to write a small 2D software render that renders the text into a texture, and sends it to the video card, as the user types.

It's not that hard:

- Store all your characters in a 1-bit bitmap.
- Check how long the message is, and create bitmap long enough to hold it
- Copy the image for each character in the message into the correct position into the new bitmap you created
- Now create a texture from that bitmap and display it on the screen.

#7 Think I shall modify gFont.cc, am I right?

for(S32 i = 32; i < 256; i++)
{
if(GetGlyphOutline(
dc, // handle of device context
i, // character to query
GGO_GRAY8_BITMAP, // format of data to return
&metrics, // address of structure for metrics
sizeof(scratchPad), // size of buffer for data
scratchPad, // address of buffer for data
&matrix // address of transformation matrix structure
) != GDI_ERROR)
{
U32 rowStride = (metrics.gmBlackBoxX + 3) & ~3; // DWORD aligned
S32 inc = metrics.gmCellIncX;
if(inc < 0)
inc = -inc;
ret->insertBitmap(i, scratchPad, rowStride, metrics.gmBlackBoxX, metrics.gmBlackBoxY, metrics.gmptGlyphOrigin.x, metrics.gmptGlyphOrigin.y, metrics.gmCellIncX);
}
else
{
SIZE size;
GetTextExtentPoint32(dc, (char*)&i, 1, &size);
if(size.cx)
ret->insertBitmap(i, scratchPad, 0, 0, 0, 0, 0, size.cx);
}
}

Change these codes to make effect?

#8 That's the code that reads the font from the system and puts it into bitmaps. It'll need sime heavy modifications, since I think it works using single byte characters. You'll need to use google or check the MSDN database to search for the code to get outlines from double-byte/unicode characters.

But you'll also have to make changes in a lot of places that handle string messages, since they all must support double-byte.

#9 First of all, you should keep that codes for single-byte character. Next, you have to generate the Chinese character code (BIG5 or GB)

The following is my modified createFont(..) in winFont.cc

GFont *createFont(const char *name, dsize_t size)
{
   if(!name)
      return NULL;
   if(size < 1)
      return NULL;
   

   HFONT hNewFont;

   if (dStricmp(name, "gfont") == 0)
		hNewFont = CreateFont(size,0,0,0,0,0,0,0,0,0,0,0,0,"&#32048;&#26126;&#39636;");
   else
	   hNewFont = CreateFont(size,0,0,0,0,0,0,0,0,0,0,0,0,name);
   
   if(!hNewFont)
      return NULL;
   
   GFont *retFont = new GFont;
   static U8 scratchPad[65536];         // YOU HAVE TO MAKE IT FROM 256 TO 65536
   static U8 scratchPad2[65536];

// OMITTED SINCE CODE THE SAME AS ORIGINAL
//   

// I ONLY SUPPORT CHINESE CHARACTER FOR size = 12, SINCE IT WILL USE A LOT
// OF TEXTURE FOR LARGE SIZE CHARACTER
// I ALSO USE A SPECIAL FONT NAME CALLED gfont TO DISPLAY CHINESE
// I AM SUPPORT ONLY BIG5
   if (size == 12 && dStricmp(name, "gfont") == 0)
   {
	// Double byte pre-generation....It is not a good idea to add all font here, but just for testing..
   for (U32 hi=0xa1; hi<=0xf9; hi++)
   {
	   for (U32 lo=0x40; lo<=0x7e; lo++)
	   {
		   U32 i = (hi << 8) + lo;

			if(GetGlyphOutline(
				fontHDC,	// handle of device context 
				i,	// character to query 
				GGO_BITMAP,//GGO_GRAY8_BITMAP,	// format of data to return 
				&metrics,	// address of structure for metrics 
				sizeof(scratchPad2),	// size of buffer for data 
				scratchPad2,	// address of buffer for data 
				&matrix 	// address of transformation matrix structure  
				) != GDI_ERROR)
			{
				glyphCount++;
				U32 rowStride = (metrics.gmBlackBoxX + 3) & ~3; // DWORD aligned
     			U32 size = rowStride * metrics.gmBlackBoxY;
				U32 bit_rowStride = ((((metrics.gmBlackBoxX + 15) & ~15)/8) + 3) & ~3;
				//for(U32 j = 0; j < size; j++)
				U32 spi = 0;
				U32 dsti = 0;
				for (U32 j = 0; j < metrics.gmBlackBoxY; j++)
				{
					U32 tspi = spi;
					U32 spbit = 0;
					for (U32 k = 0; k < rowStride; k++)
					{
						U32 pad=0;
						if (scratchPad2[spi] & 0x80)
							pad = 255;
						scratchPad2[spi] = scratchPad2[spi] << 1;
						spbit++;
						if (spbit == 8)
						{
							spbit = 0;
							spi++;
						}

						scratchPad[dsti] = pad;
						dsti++;
					}
					spi = tspi + bit_rowStride;
				}
				S32 inc = metrics.gmCellIncX;
				if(inc < 0)
					inc = -inc;
				retFont->insertBitmap(i, scratchPad, rowStride, metrics.gmBlackBoxX, metrics.gmBlackBoxY, metrics.gmptGlyphOrigin.x, metrics.gmptGlyphOrigin.y, metrics.gmCellIncX);
			}
			else
			{
				char b = i;
				SIZE size;
				GetTextExtentPoint32(fontHDC, &b, 1, &size);
				if(size.cx)
					retFont->insertBitmap(i, scratchPad, 0, 0, 0, 0, 0, size.cx);
			}
	   }

	   for (U32 lo=0xa1; lo<=0xfe; lo++)
	   {
		   U32 i = (hi << 8) + lo;
                                   // CODE OMITTED SINCE SAME AS ABOVE....
	   }
//		DeleteObject(hNewFont2);
	}
   }
   retFont->pack(textMetric.tmHeight, textMetric.tmAscent);
   //clean up local vars      
   DeleteObject(hNewFont);

   if (!glyphCount)
      Con::errorf(ConsoleLogEntry::General,"Error creating font: %s %d",name, size);

   return retFont;
}

#10 Thank you. I will try to solve it and then give my solution here.

#11 But I have to say, it is quite a lot of codes you have to change in order to DISPLAY, even worst if you need to INPUT.

I remember I have spended a month to make it all smooth.

#12 Thanks a lot. I have being working on it for a week already and almost(I believe) solve the INPUT part and continue to DISPLAY. I'll post message here when I have some new information.

#13 To Samme Ng:

I have read your codes and found this part:

for (U32 j = 0; j < metrics.gmBlackBoxY; j++)
{
U32 tspi = spi;
U32 spbit = 0;
for (U32 k = 0; k < rowStride; k++)
{
U32 pad=0;
if (scratchPad2[spi] & 0x80)
pad = 255;
scratchPad2[spi] = scratchPad2[spi] << 1;
spbit++;
if (spbit == 8)
{
spbit = 0;
spi++;
}
scratchPad[dsti] = pad;
dsti++;
}
spi = tspi + bit_rowStride;
}

I don't understand, will you please explain that for me? Especially "scratchPad2[spi] & 0x80"

Thanks a lot.

#14 Actually that's not my code. I just copied from normal single byte ASCII bitmap generation code.
That code is testing the bits of the character. For example a 16x16 pixel character:

...7654321076543210
0 0000000110000000
1 0000001110000000
2 0000011110000000
3 0000000110000000
4 0000000110000000
5 0000000110000000
6 0000000110000000
7 0001111111110000
8 0000000000000000
:
F 0000000000000000

It first test the highest bit, then shift the byte to right to test the next bit and so on...
So there is the code "scratchPad2[spi] & 0x80" and "scratchPad2[spi] = scratchPad2[spi] << 1". Once 8 bits tested, spi will shift to next byte and repeat until end.

#15 Samme,
Your code(winFont.cc) don't compile.I don't understand, will you please explain that for me?

Thanks in advance! And this is a great resource! Thanks so much!

#16 Actually it is just a code frag for your reference only. In the original TGE's code, it loop for almost 0~0xFF, then for each code, generate a character's bitmap, and move the bit into a large bitmap until it is full.

In my code, I added the loop for all BIG5 code, and for each character, move the bit into the large bitmap, and cause almost 50 pages in order to hold all.

So you may need to understand the original TGE code before applying my modification.

#17 Or can you show me what are the errors in compilation?

#18 I have finished simplified Chinese desplay, here is my codes.

For characters limit, some codes are ignored!

In winFont.cc, you only have to modify function createFont.cc, change:

static U8 scratchPad[65536];

to

static U8 scratchPad[65536];
	static U8 scratchPad2[65536];

Keep "for" loop for alphabet and symbols.

Then add our codes between "for" loop and font's pack:

for (U16 hi=0x21; hi<=0x29; hi++)   
	{      
		for (U16 lo=0x21; lo<=0x7e; lo++)      
		{         
			U16 i = (hi << 8) + lo;         
			if(GetGlyphOutline(            
				fontHDC,						// handle of device context
				i,								// character to query
				GGO_BITMAP,					//GGO_GRAY8_BITMAP,// format of data to return
				&metrics,					// address of structure for metrics
				sizeof(scratchPad2),		// size of buffer for data
				scratchPad2,				// address of buffer for data
				&matrix						// address of transformation matrix structure
				) != GDI_ERROR)         
			{            
				glyphCount++;  
				U32 rowStride = (metrics.gmBlackBoxX + 3) & ~3; // DWORD aligned
				U32 size = rowStride * metrics.gmBlackBoxY;            
				U32 bit_rowStride = ((((metrics.gmBlackBoxX + 15) & ~15)/8) + 3) & ~3;
				U32 spi = 0;            
				U32 dsti = 0;            
				for (U32 j = 0; j < metrics.gmBlackBoxY; j++)
				{               
					U32 tspi = spi; 
					U32 spbit = 0;             
					for (U32 k = 0; k < rowStride; k++)  
					{                  
						U32 pad=0;                  
						if (scratchPad2[spi] & 0x80)                    
							pad = 255;                  
						scratchPad2[spi] = scratchPad2[spi] << 1;   
						spbit++;                  
						if (spbit == 8)                  
						{                     
							spbit = 0;                     
							spi++;                  
						}                  
						scratchPad[dsti] = pad;
						dsti++; 
					}              
					spi = tspi + bit_rowStride;    
				}           
				S32 inc = metrics.gmCellIncX;        
				if(inc < 0)             
					inc = -inc;      
				retFont->insertBitmap(i, 
					scratchPad, 
					rowStride, 
					metrics.gmBlackBoxX, 
					metrics.gmBlackBoxY, 
					metrics.gmptGlyphOrigin.x,
					metrics.gmptGlyphOrigin.y,
					metrics.gmCellIncX);        
			}         
			else         
			{            
				char b = i;            
				SIZE size;            
				GetTextExtentPoint32(fontHDC, &b, 1, &size);
				if(size.cx)              
					retFont->insertBitmap(i, scratchPad, 0, 0, 0, 0, 0, size.cx);         
			}      
		}      
	}

#19 Hold on, not finished, yet.

#20 From 0x21 to 0x29 only covers symbols, Russian, Japanese and etcs. We still need to add Chinese.

for (U16 hi=0x30; hi<=0x77; hi++)
	{      
		for (U16 lo=0x21; lo<=0x7e; lo++)      
		{         
			U16 i = (hi << 8) + lo + 0x8080;         
			if(GetGlyphOutline(            
				fontHDC,						// handle of device context
				i,								// character to query
				GGO_BITMAP,					//GGO_GRAY8_BITMAP,// format of data to return
				&metrics,					// address of structure for metrics
				sizeof(scratchPad2),		// size of buffer for data
				scratchPad2,				// address of buffer for data
				&matrix						// address of transformation matrix structure
				) != GDI_ERROR)         
			{            
				glyphCount++;            
				U32 rowStride = (metrics.gmBlackBoxX + 3) & ~3; // DWORD aligned
				U32 size = rowStride * metrics.gmBlackBoxY;            
				U32 bit_rowStride = ((((metrics.gmBlackBoxX + 15) & ~15)/8) + 3) & ~3;
				U32 spi = 0;            
				U32 dsti = 0;            
				for (U32 j = 0; j < metrics.gmBlackBoxY; j++)
				{               
					U32 tspi = spi; 
					U32 spbit = 0;             
					for (U32 k = 0; k < rowStride; k++)  
					{                  
						U32 pad=0;                  
						if (scratchPad2[spi] & 0x80)                    
							pad = 255;                  
						scratchPad2[spi] = scratchPad2[spi] << 1;   
						spbit++;                  
						if (spbit == 8)                  
						{                     
							spbit = 0;                     
							spi++;                  
						}                  
						scratchPad[dsti] = pad;
						dsti++; 
					}              
					spi = tspi + bit_rowStride;    
				}           
				S32 inc = metrics.gmCellIncX;        
				if(inc < 0)             
					inc = -inc;      
				retFont->insertBitmap(i, 
					scratchPad, 
					rowStride, 
					metrics.gmBlackBoxX, 
					metrics.gmBlackBoxY, 
					metrics.gmptGlyphOrigin.x,
					metrics.gmptGlyphOrigin.y,
					metrics.gmCellIncX);        
			}         
			else         
			{            
				char b = i;            
				SIZE size;            
				GetTextExtentPoint32(fontHDC, &b, 1, &size);
				if(size.cx)              
					retFont->insertBitmap(i, scratchPad, 0, 0, 0, 0, 0, size.cx);         
			}      
		}      
	}