Transparent textures in OpenGL do not replace quad alpha

Question

I am trying to draw a partly-transparent texture above a coloured quad, so that the colour shines through the transparent part, making an outline. This is the code:

// Setup
glEnable(GL_DEPTH_TEST);
glEnable(GL_TEXTURE_2D);
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA);

// (...)
// Get image bits in object 't'
glBindTexture(GL_TEXTURE_2D, textureIDs[idx]); 
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);
glTexImage2D(GL_TEXTURE_2D, 0, 3, t.width(), t.height(), 0, GL_RGBA, GL_UNSIGNED_BYTE, t.bits());
// (...)

glTexEnvi(GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE);

// Underlying coloured quad
glColor4d(0.0, 1.0, 0.0, 1.0);    
glBindTexture(GL_TEXTURE_2D, 0); 
glBegin(GL_QUADS);
  glVertex3d(coords11.first, coords11.second, -0.001);
  glVertex3d(coords12.first, coords12.second, -0.001);
  glVertex3d(coords21.first, coords21.second, -0.001);
  glVertex3d(coords22.first, coords22.second, -0.001);
glEnd();

// Textured quad    
glColor4d(1.0, 1.0, 1.0, 0.5);
glBindTexture(GL_TEXTURE_2D, textureIDs[castleTextureIndices[0]]); 
glBegin(GL_QUADS);
  glTexCoord2d(0.0, 0.0); glVertex3d(coords11.first, coords11.second, 0);
  glTexCoord2d(1.0, 0.0); glVertex3d(coords12.first, coords12.second, 0);
  glTexCoord2d(1.0, 1.0); glVertex3d(coords21.first, coords21.second, 0);
  glTexCoord2d(0.0, 1.0); glVertex3d(coords22.first, coords22.second, 0);
glEnd();

The texture is black except where it has the outline of a castle; that part is transparent (and white). The expected behaviour, therefore, is a black rectangle with a green outline. What I get instead is a greenish rectangle with a white outline:

Screenshot

That's with alpha on the textured quad set to 0.5. If instead I set it to 1.0, I get back the texture with no transparency, as in (oh well, can't post the second screenshot as a link; it's at s330.photobucket.com/albums/l412/TWBWar/?action=view&current=replace10.png). With alpha of 0, I get a green rectangle. It therefore seems to me that the textured quad is being blended using the alpha value set by glColor4d, instead of the value from the texture, which is what I expected from GL_REPLACE. (I experimented with GL_DECAL and GL_MODULATE as well, but I wasn't able to get the behaviour I wanted.) Can anyone tell me how to make OpenGL use the texture, not the quad, alpha for blending?

Answer 1

Get ready for a trip down memory lane.

glTexImage2D(GL_TEXTURE_2D, 0, 3, ..., ..., 0, GL_RGBA, GL_UNSIGNED_BYTE, ...);

That 3 in there is your problem. The right part of the arguments (RGBA+UNSIGNED_BYTE) mentions what your source data type is. the destination data type is the third argument (i.e. what GL will store your texture as).

As it happens, when GL 1.0 was created, texturing was somewhat simpler. So the parameter only was required to specify the number of channels you wanted for your final texture, with the mapping

1=GL_LUMINANCE
2=GL_LUMINANCE_ALPHA
3=GL_RGB
4=GL_RGBA

I believe this use is discouraged, but as you can see, still works.

So... Your texture is stored as GL_RGB, without alpha.

The rest is simple application of the spec. A replace on a texture with internal format RGB does final_color=texture_color and final_alpha=fragment_alpha (aka the source from glColor in your case). See this man page for the full tables of texture environment based on internal formats.