highlights need to self-protect at least a little

[Kristin]

I know I posted about this a few hours ago in feature requests, but this is serious enough that this "feature" needs to be reported as a bug.

 

If Highlight files are so easily corrupted, there simply needs to be some sort of warning box or a built in safety measure.

 

When the Highlight file becomes corrupted, it doesn't just disappear. If it did, I would open a backup. By contrast, it gradually corrupts itself. As a result you can work for MONTHS without realizing something is wrong, and so you keep doing daily backups of a corrupted file without knowing it.

 

I literally back up my computer every day to try to protect all my work, but I can't read the Bible cover to cover every night to check of there is a problem before backing up. By the time the corruption is bad enough to notice it is so catastrophic the file is unusable. I am needing to go back to a highlight of last year to even find something "usable" but obviously losing almost a year of work in the process barely makes it worth it.

 

I am going to radically change how I work so I am not dependent on the highlights anymore, but I am feeling pretty burned out since I had put so much work into this file.

 

I will attach a photo showing what the corruption looks like. As you can see, I could not even place the X on the left if I wanted to. Files like this are not usable.

 

Some programs have some sort of precaution, so that if the user starts doing something, it sort of self protects. It appears, I am not a programmer, but it appears that if the file gets to large it sort of starts writing over itself, or something? I don't know.  😢

 

[cweber]

Hi Kristin, file corruption is a serious issue with Accordance. I eventually reinstalled everything to finally resolve the numerous crashes I was experiencing. I know someone who has over $30k invested in Accordance modules and every-time they search it crashes on them. That user went and purchased hard cover books and is rather despondent about the application and money spent.

 

 

[Kristin]

Hi @cweber,

Wow, that really does put it in perspective. The highlight issue has been pretty serious since I use it for classes as well as a translation tool, but thankfully highlight and note corruption have been my only problems. The program opens, does research, etc. So what you said for sure puts it in perspective.

 

That is also an interesting point that maybe the corruption with the highlight is not even about the size of the file, but rather a symptom of some sort of general corruption issue. If that is so, that is obviously something which will need to be addressed. I hope whichever department looks into this type of thing (the developers?) is able to get to the root of the issue and make the program more stable over all.

 

Take care,

Kristin

[cweber]

Accordance appears a file-based system and does not store Text/Tool information in a database. One would hope when we download a file some Checksum action is performed to verify the downloaded file was not corrupted once installed. If so, then the question becomes how does a file become corrupt if you only use Accordance to access that file? I suspect then, that the larger the volume of material you purchase and install, the increased likelihood of compounding search and highlight type of issues; at least statistically speaking.

 

[Kristin]

Hi @cweber

That theory is really concerning as it makes me worry I am putting the whole program at risk every time I download something? Hopefully @Tech Support can provide some clarity concerning what happened to that guy you mentioned.

[Kristin]

.... also, it kind of makes me wonder if the highlight corruption was not internal within the highlight, but rather the result of downloading something? For example, if I downloaded some random Bible version, it would also want to use highlights which are highlighting the whole verse, and maybe that could corrupt it? The idea that downloading material could be dangerous is very concerning, so I hope Accordance can provide some clarity.

[cweber]

My intention was not to scare anyone or fearmonger

 

The 3rd party person I cited having crashes might just be suffering a lack of local computer resources such as Memory (e.g. Windows 11 operating system will use 4-8GB just for its OS, I don't about Mac OS). A solid state drive (SSD) is also recommended with systems processing lots of files. Accordance have "minimum" recommendations, but what about as your library really grows?

 

• I believe the 3rd party person is fully reinstalling everything this afternoon to then observe if it corrects their crashing negative experience, if so I'll report back that result.

 

In this thread I am identifying my own concerns with using file-based systems because they "can be", not necessarily are, problematic.  As database technologies and 'Cloud' subscription rapidly advance these advancements are being used to replace file systems. I work as a solutions architect when I am not developing database code and cite from personal experience. My views are not necessarily shared by all, but are not ignorant either. Because Accordance Support have called out multiple times that files can become corrupt it lends weight to my hypothesis.

 

If there are points I wish to make, it is for users to review whether they have sufficient hardware resources (SSD, Memory, less so CPU), especially for opening multiple Accordance workspaces/tabs with an increasing quantity of files; and it is always a good idea to perform backups. I surmise files are possibly corrupted upon download as I doubt Accordance compiles files on the fly. I'd love a tool to perform a file validity check, performing the checksum to what is on the Accordance server(s). This way all users could eliminate "corrupt" downloaded files as the cause of crashes for them.

 

The root cause of corruption is something we'd all like to understand. If you look at the structure of "\Accordance\Modules\Texts\" for example, the library file itself is compiled and any error there is a real problem. This file path also contains an XML file to list details, yet We'd need to understand a lot more about what files are read and written to, whether they are locked during writes and so forth, before we could "analysis root cause", an analysis action I am certain would frustrate Accordance staff were we to try

 

I don't know that highlighting will contribute to corruption for your situation. I have not observed how highlights are saved to "guess" at the cause in your situation, but I would not rule out a fresh "corrupted" download triggering an issue. We really need a DEVELOPER from Accordance to weigh in here as typically technical support refer such a deep dive analysis to them anyway.

 

 

 

 

Edited September 10, 2022 by cweber

[cweber]

Reporting back, my friend has recovered Accordance and can use it again without crashes. They completely removed Accordance from every device, disabled Dropbox and syncing, turned off auto-updates, they even clear out auto-saves regularly now. Lastly, this person has adopted a tread very lightly approach to Accordance as a tool and only perform simple searches and use an iPad like a Kindle device to read only. This very reserved methodology has allowed use of the large financial investment, but not the user experience they first hoped for when purchasing the Expert series.

 

[Kristin]

Hi @cweber,

Thanks for the update. I know you thought it might be an old hardware issue with the computer, but this seems to confirm it, if I am not mistaken. I am glad to hear it is at least working in some fashion, and if they are able to get a hold of a newer computer, it seems like it is hopeful they will be able to use Accordance as it should be used.

 

Thanks again for the update, and hopefully a developer can get on the thread and answer some of the original questions, as you had mentioned above.

 

Take care,

Kristin

[cweber]

My friend and I both live in Australia but different States. It is difficult to diagnose hardware from software issues remotely and this is also true for Accordance Developers, but I'd appreciate a response to the question - do more files (installed libraries) increase the chance of file corruption (statistically seems possible)?

 

By cleaning out his installation he either bought some time until his hardware is a limitation again, or if it was software and due to corrupt files then he now has a clean install and by playing it safe will hopefully avoid that reoccurring. Hardware really is only an issue if your low on memory, sharing the small capacity with multiple applications.

 

As I tell my wife, I sold my motorbike to buy Accordance libraries   Therefore, I purchased the Discovery series and also have a large enough set of installed files to observe crashing with. I appear to have a stable installation these last many weeks and I do use multiple workspaces and challenge the Accordance capabilities. It was not always this way. My early experiences with Accordance were numerous crashes every day. I believe I had file corruption and had to reinstall everything a couple of times, after each crash was not preventable. However, I also have significantly more hardware resources. If I had to pick a crash cause I would first suggest it is file corruption because Accordance is built on an aging technology that is susceptible to that, but I hope for a hardware cause because it is easier to fix.

 

[Kristin]

Thank you for the explanation, and that is pretty intense if you actually sold a motorbike to buy Accordance. I push the software pretty hard too, so I am grateful it does not crash too often. Just sometimes.

[cweber]

1 minute ago, Kristin said:

that is pretty intense

My wife tells me I use this motorcycle excuse all the time, so it seems to have a lot of mileage  

 

In all seriousness, after my brother was murdered, some time ago now, I realised I was another of those Christians not living fully for Jesus as I should. Tragic events tend to make or break faith.  I was blessed and God carried me through. Since then I have been on a journey to change my life, so selling my bike was not as difficult as I thought.  It was my older brother attending Bible school who pointed me to Accordance. I'd heard of Logos but never liked it, too bloated.  Additionally, I've been blessed with the opportunity to witness to an Irish work colleague and Accordance has helped me process theology I previously did not pursue. So I am grateful for the product, even though I might appear critical of Accordance. As someone immersed in new technologies I cannot help myself speaking up so I hope others don't judge me to hard for that

 

I am not an academic in the sense of attending Bible school. I expend too much brain capacity during my work hours. I wonder how Paul managed with working on Tents (from memory) to pay his way and all hours teaching and preaching; there's a dedication to aim for?

 

 

 

[Kristin]

Hi @cweber,

I am so sorry to hear about your brother. That sounds really traumatic. You are for sure right about how tragic events can make or break faith. I also try to have that view you mentioned, that while what happened was horrible and tragic, you were able to know God in a deeper way as a direct result.

 

I am fortunate that I have not experienced a murder of someone close, but I do my best to keep that perspective you have about the value of tragic events when it comes to trauma I have gone through. As hard as it was (and still is), I try to remember that if it hadn't have happened, I never would have had the audacity to question things I did. But as a result of doing so, I believe I know God in a way I never could have otherwise.

 

On a less somber note, I also agree with what you said about Accordance. Sometimes things can be frustrating, such as highlight and note corruption, etc, but I still love Accordance and literally use it hours every single day.

 

I am sorry again for the loss of your brother.

 

Take care,

Kristin

[cweber]

15 minutes ago, Kristin said:

I believe I know God in a way I never could have otherwise.

Wholeheartedly agree

[OSchrock]

On 9/9/2022 at 7:46 PM, Kristin said:

If Highlight files are so easily corrupted, there simply needs to be some sort of warning box or a built in safety measure.

...

I am going to radically change how I work so I am not dependent on the highlights anymore, but I am feeling pretty burned out since I had put so much work into this file.

 

Thank you for bringing this up and having such a good spirit about the issue! My Notes file got corrupted in 2018 and it took me a long time to get over that. I use Highlights and Notes extensively and just the thought of losing all that work makes me sick! 

Here's to hoping for better syncing and file stability soon! Maybe in Acc14?

[rcdeacon]

1 hour ago, OSchrock said:

 

Thank you for bringing this up and having such a good spirit about the issue! My Notes file got corrupted in 2018 and it took me a long time to get over that. I use Highlights and Notes extensively and just the thought of losing all that work makes me sick! 

Here's to hoping for better syncing and file stability soon! Maybe in Acc14?

Amen!

[Lawrence]

I got Accordance a couple of years ago, and while their support staff is wonderful, the application has been unstable from the time I bought it. The latest problem doesn’t even allow me to fully open the application anymore. It just sits there opening huge numbers of easy-install windows and never gets to the workspace itself.

 

With software, there’s often a significant revision of the code when the major version number goes up. I’m hoping for a stable system with the move to version 14.

 

Downloading resources in itself should not cause any problems, but the symptoms I’ve encountered suggest a bit of carelessness on the part of the developers with the little files that store user notes and customisations. There’s not much we can do about it as users - your daily backup routine is already above and beyond the norm. The next step would be to record your notes outside Accordance, but I understand that’s not practical with highlighting.

 

On the positive side, I hope all that unplanned spaced repetition is at least helping with memory and recall.

[Kristin]

Hi @Lawrence,

Did you start with Accordance 13 or Accordance 12? I might be mistaken, but I think my note and highlight problems started with Accordance 13. I might be wrong, but I don't remember these issues with Accordance 12. I wonder if there was a kink in version 13 which could be mitigated with Accordance 14 reverting to some stuff how Accordance 12 was?

 

Those of you who have had note or highlight corruption, do you remember if it started with a specific version? It might be  helpful for Accordance to know for diagnostic purposes.

 

Kristin

[Kristin]

@Tech Support,

Is there a way to tag a developer to this thread? As a few people mentioned above, we really need a developer's input on some of these corruption issues.

 

I remember I had a highlight file which was really small get corrupted (1.5 MB), so the corruption issue does not appear to be exclusive to the size of the file. Is Acc 14 potentially addressing the stability issues, or could it potentially make it worse?

 

I know I had said I was giving up on Highlights after the third major corruption, but I just really need them, so I am starting my 4th highlight file.... this time splitting the OT into three independent files. This is obnoxious, but worth it if it prevents corruption, but like I said, one of my corrupted files was small, so I am pretty nervous.

 

Thank you for any help anyone can provide.

 

Sincerely,

Kristin

[Lawrence]

@KristinI think I started using Accordance in 2020, and I'm pretty sure they were already at v.13.

 

I haven't done much systematic highlighting or note-taking in Accordance. Even custom settings (order of resources / preferred Bible / etc) are lost from time to time when I need to re-download the lot. I'm not convinced the system is stable enough to retain notes that I want to keep.

[Kristin]

Hi @Lawrence,

That is really concerning. I had thought that it was just connected to Notes and Highlights, as I have not had the other issues you mentioned. I really don't know enough about the technology behind Accordance to know if it would be connected or not. Have you reported your issues to Tech Support yet?

[Lawrence]

Hi @Kristin,

Yes, I raised the issue in "Bug Swatters - Windows" under the title "Is there some way of using Easy Install without Accordance crashing?".

It's not that the preferences start to rot on their own. When Accordance crashes, there may have been things it meant to update in the preferences file that it can't do now, leaving the file corrupted.

Don't worry. If you backup your preferences after customising the app, you should be fine on this issue. Before backup, I'd suggest quitting Accordance first (normal exit, not force-quit) so that it has a chance to tidy up the preferences file properly.

[Lawrence]

Hi @Kristin,

I'm now curious about what can be recovered. Now that we have a bit of visibility on the file format, I wonder whether the corruption is just in the metadata or in the actual data. If it's just in the metadata, you might be able to salvage some of your actual data. R. Mansfield's comment about file sizes being supportable up to 2GB are consistent with the format definition shown recently.

 

You can try for manual recovery by making a copy of a corrupted notes file, then changing its file extension (the '.acc2' or '.doc' part) to '.txt'.

Open this copy in a text editor (TextEdit / Word / etc) and see if there's anything worth salvaging.

You might need to scroll horizontally if it's all gone onto 1 line.

 

[Kristin]

Hi @Lawrence,

Thank you for the idea. I am beyond exhausted, so I need to do this later, but when I have a chance I will try to duplicate the file and change it. My recollection is that it is not that simple, as several of us have been trying to export Notes for while, but I will try it. (If anyone else in the meantime knows what happens if you turn a Note to a doc file, please comment to this).

 

Also, if you do not mind, what is the difference between "data" and "metadata"? I think the data is what I write, and the metadata is what I do not see?

 

Kristin

 

[Lawrence]

Hi @Kristin,

No problem.

Data is the stuff you type. Metadata is the information that describes your data. E.g. how many characters in your note, how many notes in the file, etc.

Accordance holds quite a bit of metadata in the note file, and if, for example, it glitched and updated the number of notes to zero, then even though the data is still all there, it would never show it because "number of notes in this file = 0" makes it think there aren't any notes to show.

It's a long shot, but I'm currently stuck for one more day due to covid rules in my area, so I thought I'd try to be helpful here.

 

Changing a note to a doc file by changing its file extension doesn't change the file. It just signals to apps that the file is of a particular form. By making it txt (text file), it signals that you want to look at the characters in the file without interpreting any of the characters as metadata. Metadata might show up as wingdings or plain text, but you can skip that and try to find your own salvageable data.

 

If you do find salvageable data, copy it out to your favourite word processor. You're probably not going to fix the notes file itself this way. Best to just grab what you can and start afresh.

Edited October 4, 2022 by Lawrence