Post Processing Tasks

Once all of the content has been moved into the database, the compiler can be used to perform post processing tasks on the data in the database. These tasks include verifying that all hypertext screen id are present in the database, verifying that all bitmaps called by the application are present on the hard drive, and building pre-defined lists of particular types of database screens.

One of the more interesting post processing tasks is building a full text search index for the text in the database. Our front-end application uses Index Applications Incorporated's Fast Text Search (FTS). This library allows efficient searching of large amounts of text. It is many times faster than using the database's own searching facilities.

The routine below illustrates how a full-text index is built. The text that we want to build an index for is in the Text field of the Screen table. Because the Text field of the Screen table can contain embedded bitmap and other data as well as text, the routine extracts only the text content of the field and stores it in the SearchText field of the Screen table record.

Sub Build_Fts ()
  Dim ds1 As dynaset

  'Create a new index file
  nftshnd = FtsCreate(sftsfile, 20, 3, 1, 1)

  'Open the new index file
  nftshnd = FtsOpen(sftsfile, 20, FTS_EXCL)
  If nftshnd < 1 Then
    <error trap>
  End If

  'Set up first and last caluses of SQL queryt
  sSearch1 = "SELECT * FROM Screen WHERE Viewer = '"
  sSearch2 = "' ORDER BY ScreenId;"

  'Create a dynaset of all screens to index
  Set ds1 = gdbRunt.CreateDynaset(sSearch1 & VN_RTF & sSearch2)

  'first screen
  ds1.MoveFirst
  
  'Process all screens in dynaset
  Do Until ds1.EOF

    If ds1("Text").FieldSize() <> 0 Then
      'get screen record for atx control
      s = ds1("ScreenID")
      Read_ScreenID s
      'Put text in atx to strip it 
      Put_Text gatxRtf, CHUNK_TEXT
      'add title
      s = ds1("ScreenTitle") & BLANK & gatxRtf.Text

      'Get FTS integer key
      lftskey = FtsAdd(nftshnd, s)
      If lftskey > 0 Then
        ds1.Edit
        ds1("SearchText") = s
        ds1("Fts") = lftskey
        ds1.Update
      End If
    End If
    ds1.MoveNext
  Loop

  'free up memory
  i = FtsClose(nftshnd)
  ds1.Close
End Sub

The routine cycles through each record in the screen table and passes the contents of its SearchText field to the FTS DLL. FTS adds an entry to an index file it creates for the text passed and returns an integer. This integer is a key that FTS will use to identify the chunk of text later if the chunk contains a search phrase. The routine saves the integer key in the Screen table record.

The core of the routine is FtsAdd call. All of the other code Prepares the parameters for this call. (nftshand is a handle to the FTS index file. s is the unformatted text version of a Screen record Text field) and stores the return value (lftskey is a long integer that identifies this screen record from the perspective of FTS).