managing vast quantities of user data

[Kristin]

This is a question directed at those of you who need to manage vast quantities of user notes. I was working with Tech Support and discovered that converting my notes to a PDF is not super practical as the first 100 pages only go to Gen 17. Billy then suggested that I ask people in the forum how to manage large quantities of data.

 

With that as a preface, I have been primarily doing my research through creating notes, as I love how they work in parallel, etc, etc. However, I have run into an issue of my notes file becoming so large that it is starting to become corrupted forcing me to urgently split the notes into numerous notes files.

 

However, this is really impractical since there is no good way to split the notes. If I divide notes by years it keeps the most recent material at the forefront, but forces me to guess the year if I want to search for a topic in a note file. Likewise, if I divide notes by book, it would become unclear where to put notes. Topics such as "peace" or whatever could be attached to multiple books. If I just copy the same note to each new note file for each book, that would solve the problem of searching, but obviously each note file would just become a duplicate and become just as large.

 

I thought of Tools, but the problem with that is I would need to know the information is needed and open the Tool, as opposed to Notes which run in parallel.

 

So.... Those of who who work with a large amount of user data, how do you manage it?

I appreciate any suggestions anyone has.

 

Sincerely,

Kristin

[Tim Hall]

Kristin,

 

I don't deal with a large amount of user data, but I did have a thought while I was reading your post. Perhaps a solution would be to make several kinds of notes file. For example a notes file for the Pentateuch, Gospels, etc. and then have a topical note file or two as well for things like 'peace', 'love', 'kingdom', etc.

 

I know I have my normal notes which I typically use for sermons, Bible studies, etc. (these are my thoughts on verses not much research goes in it). Then I have a School notes, Illustration notes (to keep track of when i use sermon illustrations), etc.

 

Not sure if this will help but wanted to share my thought.

 

Blessings!

[Kristin]

Hi @Timothy Hall,

Thank you for your thoughts. I appreciate it. I think what you suggested is likely my best bet. My main concern is that I am going to end up with hundreds of notes files. While I like your idea of a note of "Pentateuch," for example, as it sounds very organized, it would not work on this end given how large my notes are. I know I had said Gen 17 above, but it is actually only Gen 17:5. So my notes on Gen 1:1-Gen 17:5 is 100 pages, which is the largest size for a PDF. (Mac restrictions, not Accordance). Just Genesis alone is likely over 300 pages.

 

On the other hand though, even though a "Genesis" folder would be a few hundred pages and too large for a PDF, it would still be better then how it is now, and I really urgently need to do something since the file is literally getting corrupted. Thank God it is only destroying non-biblical books so far but I don't want to test my luck.

I think if I do one note per book, and then a few important topical notes, I will end up with about 70 or so notes (not counting the other random note files I already have, which I think are about 25 or 30)... but I think all things considered it is my best bet.

Thank you again for your thoughts. I appreciate it.

Sincerely,
Kristin

[ukfraser]

Kristin, user notes is the one area of accordance ive never found works for me. Principally i think in passages and topics and and linked passages and not verses. 
I thought stacks would fill the gap but that is only useful on the mbp and not on the ipad.

 

early on i kept my notes in apples notes so they sync between ipad, phone and mbps and ive maintained that and the important ones i move to word as i still prefer that to pages. that way i can still file my notes in a structure of directories and folders that make sense and ive built up over many decades.
 

Evernotes was in my workflow when on windows and android, but i moved to notes when i moved to apple. That was also my work method before I got accordance using books and then early bible software that didnt have note taking facility. so i probably migrated back to it without giving accordance enough of a chance but we all only ever use parts of software and there is enough other bits in accordance to justify the upgrades. 
 

Notes isnt perfect and some people use a full word processor but its worth experimenting with various workflows and sees what works for you.
 

;o) 

Edited September 9, 2021 by ukfraser

[Λύχνις Δαν]

Hi Kristen,

 

  As I said in another post recently I don't use Accordance User Notes really. Now in fairness I don't take notes to anywhere near the degree that you do either. I am also sympathetic to the point ukfraser makes concerning the verse as a unit of consideration. I have commented in the forums before on the need to be able to make notes on a range of verses and so on, before now.

 

  When I need to assemble disparate data from many sources on a topic, not necessarily biblical either, I tend to use Zotero. I write notes or studies up in a wordprocessor. Zotero supports note annotation related to URLs and such but you can index and search documents from various sources. There are a number of such research tools, Devonthink, Mendeley and Zotero being just a few. Given the amount of information you are working with you might consider a tool like this for searching and a capable writing tool for the notes themselves. There are many to consider and many features to consider too. In particular you will want to be careful that it provides decent support for Greek and Hebrew if you want to include such in your notes. But using a tool like these can free you from trying to put everything in one file, though of course it brings with it the inevitable question of "if not one then how many and on what basis" but that's also probably unavoidable at some point.

 

  That said, there is the issue of the integration that User Notes provide with the text. This is perhaps the strongest argument for using them. You'll want to consider how much you need that. Of course accord:// links in documents may help you so that you can easily get Accordance to bring up a passage from an external reference.

 

  This is something of a grab bag of somewhat random thoughts on the matter. You'll need to examine your workflow and the pros and cons of a substantial change to a very basic element of it. Such a change is time-consuming and you obviously don't want to have to do it again 6 months down the road. So some experimentation and research is unavoidable despite being a distraction from your main task.

 

Thanx

D

[Solly]

Kristin, do you have any acquaintances using a robust Electronic Document Management System—think medical field and business field with government and legal requirements for storage, access, and protection of documents. There are open source and commercial systems available that will handle your documents in a manner that will meet your needs much better than any note management I have ever found with any Bible Software program. Your documentation appears to have moved far beyond the casual, and methods of dealing with serious document management are well established.

 

Best wishes,

—Joseph

[Joe Weaks]

For me, text files. I keep all my notes in text files. The caveat is I don't take notes on iPadOS or iOS. There are apps for that solution, tho.

I can copy as citation out of Accordance to paste in my file and I can keep the Accord:// link if it helps.

 

Whether you're using Accordance notes, Apple Notes, or something else, you have a dynamic database, meaning more than one layer of organizational information.

Namely, the same paragraph can be associated with a Book:Chapter:Verse identifier, and a Topic: identifier. Without moving to a relational database, which is too much imho, I'd suggest adopting a standard tagging hack for your topics. Meaning: You have a note per book, and take your notes in chapter:verse order, but whenever you are writing a paragraph that fits one of your topics, just tag it with #peace# or whatever. That way you can read along in the text order, but you can also search and look at all paragraphs tagged with #grace#, etc.

[Kristin]

Hi guys,

Thank you all for your thoughts. This has been a very complicated process and I appreciate your input. It has also been surprising to me to see that no one seems to really be using Notes like I do. 

 

@ukfraser, I was really surprised to hear you use Apple notes, as the only thing I have successfully been able to do with Apple Notes is make a grocery list. If I am understanding correctly though, it sounds like you are truly using Apple notes for “notes” while I am using notes more for academic research. So apparently just like I did with the Accordance highlights, I pushed this aspect also to the brink until needing to find a more practical option. Regarding Pages, I actually agree with you that it has some issues. I thus use OpenOffice which has worked well (not for Accordance, just other work stuff). When you wrote, "the important ones i move to word as i still prefer that to pages. that way i can still file my notes in a structure of directories and folders that make sense and ive built up over many decades,” I think that is really the key, that I need a more practical method of directories and sub-directories. It occurred to me that I likely need to use more Accordance “Tools” as that has that option. So I will use Tools until I push that to the limit too, like I did with Highlights and now Notes, hopefully not, but I am seeing a pattern with my work habits. 

 

In order to create directories and sub-directories, you would need independent folders on your computer which cannot be accessed in one document, which is a problem, and again goes back to the advantage of the Accordance “Tools.” I think you are also correct that I need to experiment and figure out what works well. 

 

@Λύχνις Δαν,

I remember the thread about notes being attached to multiple verses, which would be really helpful, as part of how this got out of control is needing to link notes to numerous verses. Thank you for the other programs you mentioned. I have never heard of Zotero or the other programs you listed, but I think you raise a good point of the need for an index. While doing some translational work I have learned that Mac Numbers has been a good way to organize some material as an index, and maybe I need to look into that more. While Numbers is of course not in the same category as the programs you mentioned, I think the fundamental idea of an index is needed. This is going to be a lot of work but it really needs to be done as I literally have thousands of pages if I printed my work. I know this since a few months ago I actually tried to print some tools I have to try to organize it and was surprised to see how many pages were involved. So then I stoped and tried to organize it in the Accordance program itself without printing. In any case, this data really needs an index. On a side note, Mac Numbers does support Greek and Hebrew without a problem.

 

When you wrote, "That said, there is the issue of the integration that User Notes provide with the text. This is perhaps the strongest argument for using them,” I agree completely and it is why I started using Notes in the first place, and will keep doing so. 

 

The more I am thinking about the suggestions on this post the more it is occurring to me that I really have two fundamentally separate goals which really need to be addressed separately. First, I have a tremendous amount of academic research which needs to be organized as such, but not necessarily linked to texts. Second, I have work related stuff where I might be teaching something in a class and want to read a text, in which case that does need to be linked. I think maybe I need to somehow categorize my work in these two sections, but it is not obvious how. I need to give this some thought, but I need to figure something out fast now that I know it is possible for the notes to become corrupted if they become too massive.

 

Thank you again and you are correct that as time consuming as this is, I really need to make sure I do it correctly so that I don’t need to re-do things in 6 months.

 

@Solly,

No, I don’t have any experience with programs like you mentioned, but I think that is a very good idea to look into something like that. When you wrote, "Your documentation appears to have moved far beyond the casual, and methods of dealing with serious document management are well established” I think that really hits the nail on the head. The Accordance “Notes” are meant for “notes” and I have been using it to write a thesis. I need to sit down and make a detailed list of all of the needs I have, prioritize those needs, and try to find something which is actually designed for that purpose.

 

@Joe Weaks,

When you wrote, I keep all my notes in text files.” What type of text files? Do you mean something like Word or Pages?

 

I am intrigued when you wrote, "You have a note per book, and take your notes in chapter:verse order, but whenever you are writing a paragraph that fits one of your topics, just tag it with #peace# or whatever. That way you can read along in the text order, but you can also search and look at all paragraphs tagged with #grace#, etc.” but I don’t quite understand it. Do you mean that I have a note on a note file attached to a verse, but instead of a comprehensive note, it just has a word or two, like #grace or whatever, and that is a key for me to go to a separate document, like OpenOffice, where I have notes on that? Am I understanding this correctly?

 

Thank you all for your help.

Sincerely,

Kristin

 

[Brian W. Davidson]

I've never found notes (on any bible software platform) to be robust, flexible, and dependable enough to use extensively.

 

I use Ulysses for scripture notes (and most any other long form writing).

[Joe Weaks]

46 minutes ago, Kristin said:

 

@Joe Weaks,

When you wrote, I keep all my notes in text files.” What type of text files? Do you mean something like Word or Pages?

 

I am intrigued when you wrote, "You have a note per book, and take your notes in chapter:verse order, but whenever you are writing a paragraph that fits one of your topics, just tag it with #peace# or whatever. That way you can read along in the text order, but you can also search and look at all paragraphs tagged with #grace#, etc.” but I don’t quite understand it. Do you mean that I have a note on a note file attached to a verse, but instead of a comprehensive note, it just has a word or two, like #grace or whatever, and that is a key for me to go to a separate document, like OpenOffice, where I have notes on that? Am I understanding this correctly?

 

Thank you all for your help.

Sincerely,

Kristin

Kristin,

My notes are all .txt files. Text documents. I used TextEdit and BBedit to work on them. Have used other things in the past.

 

My suggestion to you, was to consider a tagging scheme to identify topics. So, from my understanding, right now your notes are in "User Notes", meaning they are tied to a book:chapter:verse. That's super convenient when you're working on Habakkuk 5 and want to see what you've noted prior. But your problem is that you also want to be able to find places where you wrote about topics such as peace, love, joy, etc.

A solution is simply to adopt a uniform tagging system, where you put a tag in the paragraph when a topic comes up. Such as: #hope

That way, you can search all your notes for #hope and see those hits. This allows you to keep all your content in your notes ordered by book:chapter:verse, but also allows you to jump to places where you've tagged that paragraph with a subject tag.

 

[Kristin]

3 minutes ago, Joe Weaks said:

A solution is simply to adopt a uniform tagging system, where you put a tag in the paragraph when a topic comes up. Such as: #hope

That way, you can search all your notes for #hope and see those hits. This allows you to keep all your content in your notes ordered by book:chapter:verse, but also allows you to jump to places where you've tagged that paragraph with a subject tag.

 

Hi Joe,
Thank you for the # idea, Joe, I think I finally understand what you mean and that is a good idea. I am still giving a lot of thought to everything everyone has written so far as it is a multifaceted issue. I think the main problem is what I wrote above about there being two different fundamental goals, of organizing research and class notes. I think I am going to need to incorporate several ideas, including the # system, and index, and potentially one of the programs which Λύχνις and @Brian W. Davidson mentioned above.

[mgvh]

You might also consider the (former) BibleWorks way of organizing notes.

There was a Notes directory folder, and there was a subdirectory folder for every book of the Bible.

For comments related to the book, they went in the appropriate folder.

For comments attached to a verse, BW created a separate file that went in the book's subdirectory. E.g., 1_1.xxx; 1_2.xxx, ...

Fortunately, BW note files were RTF files, so they were easy to read in any word processor.

I imported all my notes into Accordance (a helpful feature that Accordance automated), and now write all my notes in Accordance, but I am concerned to hear about the file getting too large. I also wish I could easily look at my notes outside of Accordance.

[Kristin]

@Joe Weaks,

Just FYI the # is apparently not supported by Accordance. I will attach a pic of what happened when I tried.

@mgvh,

Given that BW died, I suppose you are suggesting the folder and subfolder idea in Finder on my Mac? Yes, it is very concerning about stuff getting to large in Accordance. I spend months and months and months on a highlighting scheme in Accordance and the whole thing got so corrupted I needed to literally start over. It was discouraging, but in the end it worked out better since what I had been using the highlights for is what I really needed to be doing in a User Bible.

 

The point is, however, the Highlights got corrupted, now the Notes did, and I am concerned if the Tools will too. I am very concerned which is why I wanted to export everything to PDFs, but I realistically can't since my data is so large it drastically exceeds the 100 page PDF limit. I also agree 100% that it is a problem not being able to read notes and such outside of Accordance itself. 

[R. Mansfield]

For symbols such as the hashtag (#) you have to use a period first. Try this search to do the above: .# vengeance

 

Note the period before the hashtag and the space before vengeance. 

[Kristin]

Hi @R. Mansfield,

Thank you for clarifying and that worked. It added sort of odd. Even though there was literally only one hit with this odd .# it took a solid 5 minutes to search all my books (I had it set to "All Tools" but it searched books as well, which I am not sure is typical or not). Anyway, 5 min later it did in fact come back with the one result. I think this won't be a good quick reference though, given this, and I need to keep thinking.

On a related note, do you think Notes can be added to Research anytime soon?

 

Thank you again,

Kristin

 

[Joe Weaks]

I did not know about this serious limitation of User Notes; is there really no way to search across multiple notes? Since User Notes are not included in Research, making them unsearchable across multiple User Notes at once, and since your content makes a single User Note so large it keeps getting corrupted, I don't see how you should stay with User Notes.

Tagging a word with #mytag in a User Note does work, when searching in that Note. That demonstrates how User Notes and User Tools are saved in a very different format.

Because Accordance makes every word in a User Tool into a Field, separate from punctuation, you'd have to tag topics in Tools with a prefixed character that Accordance considers a letter and part of the word. A tag prefix could be an unusual letter xvengeance or a number 0vengeance or an odder letter such as Option-O øvengeance etc.

 

[mgvh]

This thread has gotten me a bit nervous about my notes...

I opened my main note file.

I used CTRL-A to select everything.

Right click > Copy > ... and I get an error message popup:

No more than 500 verses (or paragraphs in a tool) can be copied at one time. In order to export all of the selected text, use the "Save Text Selection" menu item.

 

I can't find the "Save Text Selection" menu item anywhere. Anyone know?

[ScottDF]

@mgvh, it should be in the File menu, 

 

 

[mgvh]

3 minutes ago, ScottDF said:

@mgvh, it should be in the File menu,

Well, that's odd...

If I open my notes by clicking on a note icon from a verse and then select all, I do have the option to save text as in the file menu.

BUT

if I open my notes in a new tab from my library, select all, then the file > save as text option is not available.

[ScottDF]

That sounds like a B-U-G. 

[Kristin]

@mgvh and @ScottDF,

I figured out "how" to do it, but it fundamentally does not work. If I go to File --> Save Text Selection --> Plain Text it "technically" works and creates a text file. HOWEVER, if you open the text file it ONLY includes the title of the Tool and nothing in it. I didn't try it with notes yet, but that is how Tools work. So....

[mgvh]

17 minutes ago, Kristin said:

@mgvh and @ScottDF,

I figured out "how" to do it, but it fundamentally does not work. If I go to File --> Save Text Selection --> Plain Text it "technically" works and creates a text file. HOWEVER, if you open the text file it ONLY includes the title of the Tool and nothing in it. I didn't try it with notes yet, but that is how Tools work. So....

I saved as RTF file, and it is preserving everything.

[mgvh]

Hmm.... When I open up my notes from clicking on the icon in the Bible text, I can save all my Bible verse connected notes.

BUT, when I open up My Notes separately, it includes notes I have attached to other non-biblical resources. Apparently that's why the Save Text As option does not work.

I have to open the other documents separately and open My Notes files from there to be able to see / copy / save them.

[Kristin]

@mgvh

Could you please explain that again? It worked with a Tool, so I went to my Notes and it was grayed out. So I went to a small note file, but that was still grayed out. It seems like you understand the reason why, could you explain this?

[mgvh]

23 minutes ago, Kristin said:

@mgvh

Could you please explain that again? It worked with a Tool, so I went to my Notes and it was grayed out. So I went to a small note file, but that was still grayed out. It seems like you understand the reason why, could you explain this?

I have a note attached to Gen 1.1. With the Bible text open, I click on the note icon attached to the verse. That opens My Notes file. I click in the note file, and select all (using CTRL-A). Now from the File menu > Save Text Selection > RTF

That works at saving a well-formatted file that I can open in MS Word or any word processor and has all my notes that are attached to a biblical verse.

It does not, however, include the notes I've attached to non-biblical texts.

 

E.g., I have some notes attached to passages in the Apostolic Fathers. I have to open that module, click on the note icon, and save all my notes for that work. I have similar notes for a few other resource modules.

 

When I click on my Library icon, scroll down to My Stuff > My Notes, I can open my notes from there which appears in its own tab. It shows me all my notes in all my resources. I am not, however, able to save all the notes from that tab, because the File > Save Text Selection option is grayed out and not available.