The mouse-hover and selection boxes in a Flex TileList only have styles for changing the colors. If you want more advanced things like changing the shape or having a border you'll need to extend the TileList class and override two methods.

drawSelectionIndicator - Draws the box around the currently selected item.
drawHighlightIndicator - Draws the box for the mouse-over item

public class RoundedSelectionTileList extends TileList
{
   public function RoundedSelectionTileList()
   {
       super();
   }

   override protected function drawSelectionIndicator(
      indicator:Sprite, x:Number, y:Number,
      width:Number, height:Number, color:uint,
      itemRenderer:IListItemRenderer):void
   {
      var g:Graphics = indicator.graphics;
      g.clear();     
      g.beginFill(0x3f4c6b);
      g.lineStyle(2,0x6e7c9d);
      g.drawRoundRect(x,y,itemRenderer.width,height,15,15);
      g.endFill();
   }

   override protected function drawHighlightIndicator(indicator:Sprite, x:Number, y:Number,
           width:Number, height:Number, color:uint,
           itemRenderer:IListItemRenderer):void
   {
      var g:Graphics = indicator.graphics;
      g.clear();
      g.beginFill(0x3f4c6b);
      g.lineStyle(2,0x6e7c9d);
      g.drawRoundRect(x,y,itemRenderer.width,height,15,15);
      g.endFill();
   } 
}

Before:


After:
While you're in there, take a look at the other draw* protected methods. You can change most aspects of the control with those.