User Notes -- Corrupted

[cweber]

9 hours ago, Silas Marrs said:

I do want to switch user notes to something simple and human readable and hard to corrupt

@Silas Marrs I have been working data warehouse (Synapse) a bit and been ingesting many types of files, including 7ZIP of CSV from the latest Oracle enterprise ERP system; they do provision this to prevent direct access to their database with some rare exceptions I need API for (PS: I hate CSV files). I only mention this to identify Accordance is not alone in these type of issues

 

JSON appears to be a popular format these days and is considered a replacement for XML, but I have no experience with storing large string values in these files. I do see Markdown often used in Git (version control) Readme files used all over the world, so it is popular and widely used. 

 

I do not have statistics about how susceptible any files formats are to corruption. However, I imagine the process to check and release a lock is similar. The JSON structure is fairly easy to work, is scalable and flexible with most programmers familiar with this format. I find Markdown is okay, but don't find it particularly scalable; probably more suitable for notation of raw text though.

 

I have seen a third party company (for an end product as large and expensive as Bentley MicroStation - computer aided design) deliver individual component specifications via distinct Microsoft Access database files. I do not suggest this particular approach. I earn my living with databases, but they require overhead to correctly manage and can suffer corruption, more so than files if you don't do this. A single large database would make more sense, but still subject to the complexities of maintenance and system overhead.

 

 

 

Edited September 30, 2022 by cweber

[cweber]

As my subconscious mind churns over the idea of a new format for notes, a potential issue exists about resolving images. Whether JSON objects or MARKDOWN, I think they both require a data scheme (Data URLs) to allow content to be embedded inline within documents. Base 64 Encoding is a way of encoding arbitrary binary data in ASCII text. Such encoding can make file sizes roughly 33% larger than their original binary representations, which means more data that potentially might be a problem for networking, and can take longer to process than binary data.

 

[cweber]

A cursory reverse engineering perspective of Scrivener shows extensive use of RTF files for data and XML files for managing that content. I would still suggest JSON than XML. This application uses XML for indexing, styles, versioning and <BinderItem UUID...> for file locality. RTF files in this application have never corrupted on me.

 

@Silas Marrs may I suggest that you review how Scrivener has implemented their file management and content storage, it works very well, my only caveat is to use JSON not the near obsolete XML. Of course the work load is unlikely to be as serious, but I think with well designed locking management it might work better than the current Notes experience and RTF opens up more possibilities. However, RTF template injection is something to be mindful about.

 

 

Edited October 1, 2022 by cweber

[cweber]

I am still thinking about backups of Notes and Highlights, but for now I've put together a Power Automate flow to extract file properties and create a Hash (checksum) for each Text and Tool library. The file properties give us the size and dates for additional comparing, but the main advantage is the Hash value. If the library becomes corrupted this will change and be identifiable in any comparison.

 

Output data from the flow

 

file properties

 

 

The idea is to manually or automate a scheduled run to check the library files and flag any errors. This assumes that Accordance does not write back to these downloaded files during normal operation, hence they change.  Looking at the file contents in an editor they are not text readable to us. These Text and Tool files are not flagged as Read-Only either but that does not mean they aren't treated that way. I plan to write the data to a local database and run a compare occasionally or after a crash. My backups would theoretically be designed about this strategy.

 

 

Well yes this is all a bit extreme, but as I've mentioned before on these forums I needed to get myself up to speed on new technologies for work and because crashing is such a pain I figure that makes a good candidate for the technical learning

 

Happy for any feedback...

 

Edited October 1, 2022 by cweber

[Kristin]

Hi @cweber,

Most of what you wrote above, as well as your post prior to that, is frankly too technical for me to understand. But since you would like feedback I want to mention that from the little I did understand, what you wrote seems like a good idea. I use Notes AND Highlights for technical research, and I know I am not the only one who it is pretty catastrophic for when then get corrupted. So anything concerning improved stability and corruption prevention sounds like a good idea.

 

Kristin

[cweber]

I guess what I hope to gain from the technical posts is for Accordance to think a bit more about new and innovative solutions for end-user self-help support. I'd like to see them enable us to "configure our purchased items", and a "verify that all of my content is good and nothing is corrupted" capability. I tried to demonstrate ways this can be achieved within the limited scope of what I can gleam from the installation. The two areas I trialed have been library management and file verification (related back to backups). I have successfully been able to improve on both of these areas of Accordance for myself without a huge effort. I do believe development funding in these two areas can return a return-on-investment (ROI) for Accordance. What might be achieved will depend on whether legacy architecture is improved upon.

 

New features are nice to have and for a company to continue moving forward, but technology can be a fickle beast. The customer needs absolute confidence in their purchase investment and that any concerns will be addressed.

 

PS: I am clueless about the languages at this stage   so the technical is the only area I feel that I can contribute to improving Accordance.

Edited October 1, 2022 by cweber

[A. Smith]

On 9/28/2022 at 5:10 PM, Kristin said:

Hi @A. Smith,

What you are describing about the Notes turning into Wingdings, typing over itself, etc is exactly how it is has been for me several times as well.

 

Yes, Billy is still here. He has always been really helpful and I appreciate working with him.

 

Regarding backups, I backup to TimeMachine at least once a day, but the problem is, the Highlights and Notes can get corrupted without you realizing it for awhile.

 

For example, lets say you are doing a bunch of Highlights and Notes for a while in Titus. You do your work, back up, do more work, backup. Everything seems to be going fine. However, as you are doing this Nehemiah is becoming corrupted beyond use. You can thus backup every day, every hour, it won't matter, since you are backing up the corruption.

 

In theory you can go to TimeMachine and find the last saved copy without corruption, but how can you do that? Import a backup, read the Bible cover to cover to check for corruption, and repeat until you find a non-corrupted backup?

 

Furthermore, I frankly can't find a non-corrupted copy. I went back two months and the problems are there also. I finally imported that, as this 2 month old copy is less corrupted, but obviously losing two months of work is another type of Note and Highlight corruption.

 

(On a side note, one thing I would like to mention is if you open your Notes as a parallel, and you are scrolling and as you do so the Notes start typing over themselves, don't panic. Just close the parallel and reopen it, and that will fix it 99% of the time. I don't know what is causing that, but it seems to be an unrelated, almost harmless issue.

 

Kristin

You’ve made great points and I don’t doubt you’re wrong. But this isn’t the way it’s worked for me. Thankfully, I haven’t observed notes corrupting in one part but not another such that I wouldn’t observe it. But that is terrible!