Add token_index column to _TOKENIZE table.
If the new optional fifth argument to _TOKENIZE is non-zero, the _TOKENIZE function now uses a third column, token_index, in addition to the earlier token and source columns. The new column stores the index of the token within the string. This is useful for distingusihing between matches that are prefixes of the original string, and matches that are prefixes of one of the tokens inside the original string. This change is required to fix http://b/issue?id=1847321
Loading
Please sign in to comment