Linking Documents in a Semantic Graph Database
Originally posted on Towards Data Science by Akash Kaul. Follow Akash on LinkedIn.
Part 1 of this blog series reviews how the data is extracted before being used in TigerGraph. Review Part 1 here: https://towardsdatascience.com/using-scispacy-for-named-entity-recognition-785389e7918d
1. Building a Graph Using Publication Metadata
This project is a continuation of a previous project. We used scispaCy, an NLP package for Python, to extract medical keywords from the abstracts of a collection of articles on Covid-19. Check out that post for an in-depth guide on how we extracted the data. The goal of this project is to build a graph to connect the publications to the entities we extracted. We are using a graph because it allows us to efficiently display and analyze our highly connected data. If you aren’t already familiar with graphs, I suggest checking out this video. To create our graph, we’ll be using a free, UI-based graphing platform from TigerGraph called TigerGraph Cloud.
2. Create a TigerGraph Account
Before you do anything else, you will first need to create a TigerGraph account. TigerGraph has a great video that takes you through the steps to create your account and build an empty graph. Follow along with the steps in the video, and load up your solution in GraphStudio. After the solution is loaded, you should reach this homepage.
3. Designing a Graph Schema
We’ll start by clicking the Design Schema tab. You should see a blank page that looks something like this.
Next, we’ll create some vertices. Our central vertex type will be called “PUBLICATION”. This vertex will store data related to the publication like the doi, the title, and the URL. These specific properties for a vertex type are known as attributes. To create the vertex, press the + icon. Fill in the data as follows.
There are some publication attributes, like the license type and journal of publication, that we didn’t list here. That’s because ideally, attributes are traits that will be UNIQUE for each vertex of a given type. Since multiple publications can have the same license, journal, etc., it’s best to leave those as SEPARATE VERTEX TYPES. We can still easily connect those vertices in our design schema.
Let’s continue creating our vertices. We’ll need 3 new vertices for highly repetitive publication traits (license, author, and journal), a vertex for the entities we extracted previously, and a vertex for the classes of those entities. These vertices should look like this.
With all of our vertices now created, our graph should look something like this.
It looks pretty good, but it’s missing something. We still need to connect our vertices with edges!
If you haven’t already, make sure to publish your graph using the uparrow next to the + sign. This ensures your graph is saved.
To add an edge, click the right arrow icon and select the two vertices that you want to edge to connect. For example, if we want to connect the author vertex to the publication vertex, we press the arrow, then click on the author icon and the publication icon. We should get a pop-up that looks like this:
TigerGraph automatically fills in the source and target vertex, so all we have to do is name it. It’s helpful to name the edge after the two vertices it connects. For this edge, let’s call it “PUB_HAS_AUTHOR”. Our edge should now look like this:
Now we just repeat the process for our remaining edges.
Edge between Publication and Journal
Note that all of our edges are undirected. This is because the relationship between any 2 vertices goes both ways (i.e. a publication is tied to an author but the author is also tied to that publication).
After we add all of these edges, our final graph should look something like this:
4. Uploading & Mapping our Data
With our graph design finished, we now need to upload our data. You can find all of the files that we need to upload here. I’ll briefly explain the contents of each file.
sample.csv — contains general information about each publication (like the doi, URL, journal, license, etc.
normalizedAuthors.csv — contains a normalized list of the doi and authors of each publication
Entity_*.csv — Each of these four files contain the entities and classes extracted using one of the prebuilt models in scispaCy. If you want to see how these medical terms were extracted, check out my last article.
To upload these files, click the Map Data to Graph tab and then hit the icon that looks like a file with a + sign. Now add all the files you downloaded to the server and click ADD.
This is where the tricky part begins. You’ll need to manually map each file to every vertex and edge that corresponds to a column in that file. For example, the normalizedAuthors filehas the columns “doi” and “author”. This means the file has to be mapped to the publication vertex (where the doi attribute lives), the author vertex, and the edge that connects the publication and author vertices. To map the file, click the crossed arrows icon, select your file, and select a vertex or edge. The mapping for the normalizedAuthors file should look like this:
And each individual mapping should look like this:
Again, we repeat these steps for our other files as shown.
Total mapping of sample.csv
Mapping sample.csv to Journal Vertex
Mapping sample.csv to edge connecting Publication and Journal
Mapping sample.csv to Publication vertex
Mapping sample.csv to edge connecting Publication and License
Mapping sample.csv to License vertex
Now we map the entity files. I’ll show the mapping for one file since it’s the same for all four.
Total mapping for entity file
You’ve just completed the hardest part of this project. With the mapping done, our graph should look something like this:
Hit the publish button to save all of your hard work!
5. Load Data
Once the data is mapped, loading the data is quite easy. Click the Load Data tab and then hit the play button to start loading your data. This process should take about a minute. Once it’s done, you can move on to visualizing your graph.
6. Visually Exploring your Graph
Now for the fun part (although I hope the whole project has been fun thus far). With your design created and your data loaded, you can now visually explore your graph. To do so, hit the Explore Graph tab. There’s a lot you can do on this page (the full documentation can be found here), but let’s look at a couple of specific examples.
Searching for a vertex
You can search for a specific vertex using the vertex id, or you can find a random vertex of a specific type. We’ll do the latter. For our example, let’s pick 5 vertices of the type “PUBLICATION”. You should see 5 vertices pop up.
Example search for 5 vertices of type Publication
You can hover over each vertex to see all of its attributes. You can also double-click any vertex to bring up its immediate connections.
Visual after double-clicking a Publication vertex
If your output looks messy, you can click and drag vertices around to clean up your screen.
Expand from Vertices
If you change from the Search option (magnifying glass) to the Expand from Vertices option (triangle-looking symbol), you can expand from a vertex to other vertices beyond the immediate connections. To make sure your computer doesn’t overload, change the expanding edge limit to something small like 4 or 5. Now, let’s see what happens when we expand on one of the entities we just revealed.
Expand from Entity vertex through all edge types, towards all vertex types
We can see that upon expansion of our entity vertex, the class of the entity, as well as other papers that also have that same entity, are pulled up. This is a great example of why we used a graph in the first place. We can easily search our graph for publications, entities, authors, licenses, etc., and with a few clicks, we can show how different publications are connected, which publications share common entities, and much much more.
I won’t cover the other graph exploration options or how you can create queries (which are essentially code versions of the visual expansions we just used) in this article. Stay tuned for Part 3 of this series, where I will take you through writing some basic search queries for our graph. Until then, I’ll leave them as challenge problems for you to tackle on your own. If you’re interested, you can read about queries here.
Graphs offer unparalleled benefits for mapping and analyzing highly connected data. Through our exploration of TigerGraph Cloud, you have learned how to visually design, map, and explore your own fully-fledged graph. You’ve made a huge first step into the world of graph databases and gained a considerable advantage for your future projects. Graph databases can be implemented on everything from small-scale projects like this one to larger structures like Facebook and Google Search. Graphs possess incredible power; a power that you now have access to.