user bible Unable to Import User Bible in 14.0.7 (Possible Regression from 13.3.4)

[Steven S]

Environment

• Accordance 14.0.7
• Windows 11 [Version 10.0.22621.2134]

 

Description
A user Bible that can be imported successfully by Accordance 13.3.4 now fails to be imported in Accordance 14.0.7 with the error: "The HTML tag at the end of the text ... cannot be used without first setting its corresponding tag."

 

Reproduction frequency
Always.

 

Reproduction steps

1. Import the Bible text in the attached TEST.txt file (TEST.txt) with the File > User Files > Import Bible Text... command using default versification. (Note that this file only contains Genesis 1 of my user Bible, but that seems to be enough to trigger the error.)

 

Expected behavior
The user Bible should be imported without error.

 

Actual behavior
Accordance fails to import the user Bible with the error: "The HTML tag at the end of the text 'over all the earth,<sup><i>a</i>' cannot be used without first setting its corresponding tag."

 

The error occurs in verse 26 of the sample file, and it is the second occurrence of HTML tags in the file (the first being in verse 2).

 

Notes
The following posts from 2021 seem related, but, in my case, the offending tags are not at the beginning of the verse:

 

 

 

[Kristin]

Hi @Steven S,
I personally needed a lot of help from Fabian to be able to even import anything, so I am not an expert by any means, but the ^<i>a</i> seems to be the issue. Granted, the issue is not at the end as it claims, but rather, my understanding is that Accordance user bibles don't allow superscripts or subscripts. Maybe something has changed since I was taught this, but the error might be from trying to use them. Did the old user bible also have this tagging?

I think I would personally try deleting the tags from the Test and re-upload it. Regardless if that resolves the issue or not, it would be helpful for diagnostic purposes.

Kristin

[Dr. Nathan Parker]

I can reproduce this on my 14.0.7/Windows 11 system. I'm showing it to a colleague with 13 to see what happens. If we need to report it as a bug report, we can.

[Kristin]

Hi @Nathan Parker,

If this is a bug, does that mean that subscripts and superscripts are now able to be included in user bibles? Is that a recent change?

[Steven S]

Hi @Kristin. Agreed, Fabian provided me with lots of help when I began my user Bible journey last year. 

 

I use the <sup> tag all throughout my user Bible and have not had an issue on any of the 13.3.x builds (I don't remember the first version of Accordance with which I started this project, but it definitely wasn't older than 13.2.x).

 

I did try playing around with the user Bible content to see if I could reduce the reproduction text down to the minimum amount needed. For example, I altered the nested tags in verse 26 (^<i>a</i>) and tried both ^a alone and <i>a</i> alone, but received a similar error. Indeed, if I remove the tags from verse 26 completely (but leave the identical tags in verse 2), the text is imported without error.

 

But what was especially frustrating, when I tried to replace the Bible text with text from a Lorem Ipsum generator across only a few verses (but still randomly throwing in the HTML tags), I couldn't reproduce the error. It's been several hours since I was playing around with it, but I remember thinking at the time that the reproduction might require the presence of a particular character or type of character (e.g., a Unicode character with a code point greater than 0x7F so it generates two bytes or more when encoded using UTF-8) somewhere in the text before the HTML tag. I should have some time later today to fiddle around with this a bit more, and I'll post any new facts I discover.

[Steven S]

I believe I found the root cause of the error in the sample text: the comma preceding the opening <sup> tag in verse 26. If I add a space between the two, 14.0.7 imports the text successfully. I'm guessing there's some kind of pattern matching going on to find the HTML tags in the text, and the pattern has changed somewhere between 13.3.4 and 14.0.7 to now exclude a comma immediately prior to the opening tag.

 

I'll do some more thorough testing later and post information about any additional characters I find that, when adjacent to an HTML tag, cause the import to fail.

[Kristin]

Hi @Steven S,

Thanks for the updates and that all sounds good. I am just curious, now that you can use superscripts, I assume they are for footnotes. Are you able to get footnotes to show up in Instant Details now? Or are your superscripts being matched to something in a Note file?

 

Kristin

[Steven S]

1 hour ago, Kristin said:

I am just curious, now that you can use superscripts, I assume they are for footnotes. Are you able to get footnotes to show up in Instant Details now? Or are your superscripts being matched to something in a Note file?

 

The latter. I am simply using superscripts in the user Bible which correspond to an entry in a user notes file. AFAIK, based on a discussion I had with Fabian last year, the former is not possible, and I haven't seen anything in the various 14.0.x release notes that would indicate such a feature has been added.

[Kristin]

Hi @Steven S,

Ok, thanks for clarifying, and that makes sense. It was a few years ago, so I don't remember the details precisely, I just remember Fabian said  footnotes aren't possible. Given what you wrote, that now reminds me that he had told me the same thing. You can "technically" have footnotes (such as superscripts etc), BUT they don't go anywhere in Instant Details. So I supposed I translated that in my mind as not being possible at all.

[Dr. Nathan Parker]

I'm not sure on superscripts/footnotes in User Bibles (I can ask if need be). Definitely keep us posted on your findings!

[Steven S]

I searched my user Bible for all characters that appear immediately before a start tag (e.g., !^{) or immediately after an end tag (e.g.,}!) and came up with 56 unique combinations of character and position. I then created a test case for each instance to verify whether Accordance 14.0.7 could import it without error. The results were 48 tests passed, while 8 failed. All tests passed when run on Accordance 13.3.4.

 

The characters that trigger the error described in the OP all appear immediately before a start tag (i.e., no character immediately after an end tag triggered the error) and are listed below:

• ! (U+0021, EXCLAMATION MARK)
• ) (U+0029, RIGHT PARENTHESIS)
• , (U+002C, COMMA)
• . (U+002E, FULL STOP [PERIOD])
• : (U+003A, COLON)
• ? (U+003F, QUESTION MARK)
• ’ (U+2019, RIGHT SINGLE QUOTATION MARK)
• ” (U+201D, RIGHT DOUBLE QUOTATION MARK)

 

Obviously, there could be other characters that trigger the error, but, as I said above, I only used those that appear in my user Bible as candidates; I didn't want to test the entire Unicode code space. 

 

I've attached two user Bibles that can be used for acceptance testing. The first, TEST-PASS.txt (TEST-PASS.txt), contains one test for each character from my user Bible that appears adjacent to a tag and for which both Accordance 13.3.4 and 14.0.7 will successfully import. The second, TEST-FAIL.txt (TEST-FAIL.txt), contains one test for each character from my user Bible that appears adjacent to a tag and for which Accordance 14.0.7 fails to import, but 13.3.4 successfully imports. Hopefully, these test cases will prove helpful for the dev team.

[Dr. Nathan Parker]

Thanks for the info! I'm sharing this with a colleague in case we need to report this as a bug.

[simonau]

Hi @Nathan Parker

 

Just wondering did you fire a bug about the Accordance import issue which it could not handle the punctuation marks (Unicode) before and after the tag?

 

In my bible texts, those punctuation marks cause the importing fails (I'm sure there are more): 

，<

>，
；<

>；
。<
>。

、<
>、

？<

>？
！<

>！

（<

>（

）<

>）

「<

>「

」<

>」

！<
>！

This problem is in Accordance 14, but not in Accordance 13 so it is a regression.

 

I add a space in between those marks and the angle brackets as a workaround.

The documentation https://accordancefiles2.com/helpfiles/14-macOS/index.htm#t=mac_14%2Fcontent%2Ftopics%2F05_dd%2Fpreparing_the_text-ub.htm&rhsearch=import bible text specify those HTML tag are supported. It documents to support Unicode as well.

 

 

To be honest, there are rooms to improve for Accordance to maintain its world-class software in high quality.  

 

Regards,

--Simon

Edited November 27, 2023 by simonau

[Dr. Nathan Parker]

I'll grab all of this to ensure it gets filed in the report.

[Steven S]

8 hours ago, simonau said:

I add a space in between those marks and the angle brackets as a workaround.

 

That's good to know! I've been holding off on upgrading to Accordance 14 until this is fixed because my user Bibles are an integral part of my daily Bible study. However, if the fix for this regression is delayed past 14.1, I'll seriously consider creating a separate v14 branch that implements this (and possibly other) workarounds.

[Dr. Nathan Parker]

A user emailed me some additional feedback on this:

 

I reported this before, but it hasn’t been fixed. This bug is foremost with unicode User Bibles.

Also it would be good if Silas add the ability to import User Bibles with Legacy Mac (CR), Unix (LF) and Windows (CRLF) line endings. This is also still often an import issue.