Backbone of bookscraper

[must Janik]

DO NOT REVIEW IT. I PROVIDED BIG CHANGES, SO THIS CODE IS NOT UPDATED

I'm developing my bookscraper to improve myself in coding and I think I have solid structure that I want to be reviewed by more experienced people than me. It is app based on SpringBoot which fetches data from 2 different bookstores. For now it has 3 options of fetching books.

1) Bestsellers

2) Book with given title e.g: You say you want to get book with title "Great boy" and it gives you book from each bookstore with title, price, link to the book

3) Categorized books (currently for 5 categories e.g.: crimes, biogrpahies etc)

I have created ranking for categorized books which takes 15 books from each bookstore, then it counts how many given title was repeated (if for example two, then it is higher in the ranking) and returns it in the map Map<String,Integer> which goes for title and number of occurrences.

I have integration and unit tests. In my test resources I put 3 files for each bookstore which contains .html files for each type of fetching (bestsellers,categorized book,most precise book).

Here is the link to github, because I'm not gonna put all the classes here as there is a lot of them. https://github.com/must1/BookstoreScraper

I added also CI with Travis and SonarCloud.

JsoupConnector (responsible for connecting to the given url)

package bookstore.scraper.utilities;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.springframework.stereotype.Component;

import java.io.IOException;

@Component
public class JSoupConnector {

    public Document connect(String url) {
        try {
            return Jsoup.connect(url).get();
        } catch (IOException e) {
            throw new IllegalArgumentException("Cannot connect to" + url);
        }
    }
}

MerlinUrlProperties (all links are stored in .yml file), same class for second bookstore

package bookstore.scraper.urlproperties;

import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

@Getter
@Setter
@Component
@ConfigurationProperties("external.library.url")
public class MerlinUrlProperties {

    private Merlin merlin = new Merlin();

    @Getter
    @Setter
    public static class Merlin {

        private String mostPreciseBook;
        private String bestSellers;
        private String concreteBook;
        private String romances;
        private String biographies;
        private String crime;
        private String guides;
        private String fantasy;
    }
}

MerlinFetchingBookService (fetches data from the net)

package bookstore.scraper.fetcher.merlin;

import bookstore.scraper.book.Book;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

@Service
public class MerlinFetchingBookService {

    private static final int BESTSELLERS_NUMBER_TO_FETCH = 6;
    private static final int CATEGORIZED_BOOKS_NUMBER_TO_FETCH = 16;

    private final MerlinUrlProperties merlinUrlProperties;

    @Autowired
    public MerlinFetchingBookService(MerlinUrlProperties merlinUrlProperties) {
        this.merlinUrlProperties = merlinUrlProperties;
    }

    public Book getMostPreciseMerlinBook(Document document) {
        String title = document.select("div.b-products-list__title-holder").select("a").first().text();
        String price = document.select("div.b-products-list__price-holder > a").first().text();
        String author = document.select("div.b-products-list__manufacturer-holder").select("a").first().text();
        String productID = document.select("div.grid__col.grid__col--20-80-80.b-products-wrap > ul > li:nth-child(1)").first().attr("data-ppc-id");
        String bookUrl = createBookUrl(title, productID);

        return new Book.BookBuilder()
                .withAuthor(author)
                .withPrice(price)
                .withTitle(title)
                .withProductID(productID)
                .withBookUrl(bookUrl)
                .build();
    }

    public List<Book> get5BestSellersMerlin(Document document) {
        return IntStream.range(1, BESTSELLERS_NUMBER_TO_FETCH)
                .mapToObj(iterator -> getBestSellerOrCategorizedBook(document, iterator))
                .collect(Collectors.toList());
    }

    public List<Book> get15BooksFromCategory(Document document) {
        return IntStream.range(1, CATEGORIZED_BOOKS_NUMBER_TO_FETCH)
                .mapToObj(iterator -> getBestSellerOrCategorizedBook(document, iterator))
                .collect(Collectors.toList());
    }

    //Todo
    //need to think about better name for the method as it can be used for fetching bestseller and categorized book
    private Book getBestSellerOrCategorizedBook(Document document, int iteratedBook) {
        Elements siteElements = document.select("div.grid__col.grid__col--20-80-80.b-products-wrap > ul > li:nth-child(" + iteratedBook + ")");
        String author = siteElements.select(" > div > div.b-products-list__desc-wrap > div > div.b-products-list__main-content > div.b-products-list__desc-prime > div.b-products-list__manufacturer-holder").select("a").first().text();
        String title = siteElements.select(" > div > div.b-products-list__desc-wrap > div > div.b-products-list__main-content > div.b-products-list__desc-prime > div.b-products-list__title-holder > a").first().text();
        String price = siteElements.select(" div.b-products-list__price-holder > a").first().text();
        String productID = siteElements.first().attr("data-ppc-id");
        String bookUrl = createBookUrl(title, productID);

        return new Book.BookBuilder()
                .withAuthor(author)
                .withPrice(price)
                .withTitle(title)
                .withProductID(productID)
                .withBookUrl(bookUrl)
                .build();
    }

    private String createBookUrl(String title, String productID) {
        return String.format(merlinUrlProperties.getMerlin().getConcreteBook(), title, productID);
    }
}

CategorizedBookRankingService (responsible for ranking for given category)

package bookstore.scraper.rankingsystem;

import bookstore.scraper.book.Book;
import bookstore.scraper.book.scrapingtypeservice.CategorizedBookService;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.*;
import java.util.stream.Stream;

import static java.util.stream.Collectors.*;

@Slf4j
@Component
public class CategorizedBooksRankingService {

    private final CategorizedBookService categorizedBookService;

    @Autowired
    public CategorizedBooksRankingService(CategorizedBookService categorizedBookService) {
        this.categorizedBookService = categorizedBookService;
    }

    public Map<String, Integer> getRankingForCategory(CategoryType category) {
        Map<Bookstore, List<Book>> bookstoreWith15CategorizedBooks = chooseGetterImplementationByCategory(category);

        List<Book> merlinBooks = bookstoreWith15CategorizedBooks.get(Bookstore.MERLIN);
        List<Book> empikBooks = bookstoreWith15CategorizedBooks.get(Bookstore.EMPIK);

        Map<String, List<String>> purifiedTitleWithOriginalTitles = getPurifiedTitleWithAccordingOriginalTitles(merlinBooks, empikBooks);
        Map<String, Integer> bookTitleWithOccurrencesNumber = getTitleWithOccurrences(purifiedTitleWithOriginalTitles);

        return getSortedLinkedHashMapByValue(bookTitleWithOccurrencesNumber);
    }

    private Map<String, List<String>> getPurifiedTitleWithAccordingOriginalTitles(List<Book> list1, List<Book> list2) {
        return Stream.concat(list1.stream(), list2.stream())
                .collect(
                        groupingBy(Book::getPurifiedTitle, mapping(Book::getTitle, toList())));
    }

    private Map<String, Integer> getTitleWithOccurrences(Map<String, List<String>> map) {
        return map.values().stream().collect(toMap(list -> list.get(0), List::size));
    }

    private Map<String, Integer> getSortedLinkedHashMapByValue(Map<String, Integer> mapToSort) {
        return mapToSort.entrySet()
                .stream()
                .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))
                .collect(
                        toMap(Map.Entry::getKey, Map.Entry::getValue, (e1, e2) -> e2,
                                LinkedHashMap::new));
    }

    private Map<Bookstore, List<Book>> chooseGetterImplementationByCategory(CategoryType categoryType) {
        Map<Bookstore, List<Book>> map = new EnumMap<>(Bookstore.class);

        if (categoryType.equals(CategoryType.CRIME))
            map = categorizedBookService.get15BooksFromCrimeCategory();
        if (categoryType.equals(CategoryType.ROMANCES))
            map = categorizedBookService.get15BooksFromRomanceCategory();
        if (categoryType.equals(CategoryType.FANTASY))
            map = categorizedBookService.get15BooksFromFantasyCategory();
        if (categoryType.equals(CategoryType.GUIDES))
            map = categorizedBookService.get15BooksFromGuidesCategory();
        if (categoryType.equals(CategoryType.BIOGRAPHY))
            map = categorizedBookService.get15BooksFromBiographiesCategory();

        return map;
    }
}

CategorizedBookService

package bookstore.scraper.book.scrapingtypeservice;

import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.book.Book;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookService;
import bookstore.scraper.fetcher.merlin.MerlinFetchingBookService;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.utilities.JSoupConnector;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.EnumMap;
import java.util.List;
import java.util.Map;


@Service
@Slf4j
public class CategorizedBookService {

    private final EmpikFetchingBookService empikBookService;
    private final MerlinFetchingBookService merlinFetchingBookService;
    private final EmpikUrlProperties empikUrlProperties;
    private final MerlinUrlProperties merlinUrlProperties;
    private final JSoupConnector jSoupConnector;

    @Autowired
    public CategorizedBookService(EmpikFetchingBookService empikBookService, MerlinFetchingBookService merlinFetchingBookService, EmpikUrlProperties empikUrlProperties, MerlinUrlProperties merlinUrlProperties, JSoupConnector jSoupConnector) {
        this.empikBookService = empikBookService;
        this.merlinFetchingBookService = merlinFetchingBookService;
        this.empikUrlProperties = empikUrlProperties;
        this.merlinUrlProperties = merlinUrlProperties;
        this.jSoupConnector = jSoupConnector;
    }

    public Map<Bookstore, List<Book>> get15BooksFromRomanceCategory() {
        return get15BooksFrom(empikUrlProperties.getEmpik().getRomances(), merlinUrlProperties.getMerlin().getRomances());
    }

    public Map<Bookstore, List<Book>> get15BooksFromFantasyCategory() {
        return get15BooksFrom(empikUrlProperties.getEmpik().getFantasy(), merlinUrlProperties.getMerlin().getFantasy());
    }

    public Map<Bookstore, List<Book>> get15BooksFromCrimeCategory() {
        return get15BooksFrom(empikUrlProperties.getEmpik().getCrime(), merlinUrlProperties.getMerlin().getCrime());
    }

    public Map<Bookstore, List<Book>> get15BooksFromGuidesCategory() {
        return get15BooksFrom(empikUrlProperties.getEmpik().getGuides(), merlinUrlProperties.getMerlin().getGuides());
    }

    public Map<Bookstore, List<Book>> get15BooksFromBiographiesCategory() {
        return get15BooksFrom(empikUrlProperties.getEmpik().getBiographies(), merlinUrlProperties.getMerlin().getBiographies());
    }

    private Map<Bookstore, List<Book>> get15BooksFrom(String bookStoreEmpikURL, String bookStoreMerlinURL) {
        Map<Bookstore, List<Book>> bookstoreWith15CategorizedBooks = new EnumMap<>(Bookstore.class);

        bookstoreWith15CategorizedBooks.put(Bookstore.EMPIK, empikBookService
                .get15BooksFromCategory(jSoupConnector.connect(bookStoreEmpikURL)));
        bookstoreWith15CategorizedBooks.put(Bookstore.MERLIN, merlinFetchingBookService
                .get15BooksFromCategory(jSoupConnector.connect(bookStoreMerlinURL)));

        return bookstoreWith15CategorizedBooks;
    }
}

MostPreciseBookService

package bookstore.scraper.book.scrapingtypeservice;

import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.book.Book;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookService;
import bookstore.scraper.fetcher.merlin.MerlinFetchingBookService;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.utilities.JSoupConnector;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.EnumMap;
import java.util.Map;

@Service
public class MostPreciseBookService {

    private final EmpikFetchingBookService empikBookService;
    private final MerlinFetchingBookService merlinBookService;

    private final EmpikUrlProperties empikUrlProperties;
    private final MerlinUrlProperties merlinUrlProperties;

    private final JSoupConnector jSoupConnector;

    @Autowired
    public MostPreciseBookService(EmpikFetchingBookService empikBookService, MerlinFetchingBookService merlinBookService, EmpikUrlProperties empikUrlProperties, MerlinUrlProperties merlinUrlProperties, JSoupConnector jSoupConnector) {
        this.empikBookService = empikBookService;
        this.merlinBookService = merlinBookService;
        this.empikUrlProperties = empikUrlProperties;
        this.merlinUrlProperties = merlinUrlProperties;
        this.jSoupConnector = jSoupConnector;
    }

    public Map<Bookstore, Book> getBookByTitle(String title) {
        Map<Bookstore, Book> bookstoreWithMostPreciseBook = new EnumMap<>(Bookstore.class);

        bookstoreWithMostPreciseBook.put(Bookstore.MERLIN,
                merlinBookService.getMostPreciseMerlinBook(
                        jSoupConnector.connect(concatUrlWithTitle(merlinUrlProperties.getMerlin().getMostPreciseBook(), title))));

        bookstoreWithMostPreciseBook.put(Bookstore.EMPIK,
                empikBookService.getMostPreciseEmpikBook(
                        jSoupConnector.connect(concatUrlWithTitle(empikUrlProperties.getEmpik().getMostPreciseBook(), title))));

        return bookstoreWithMostPreciseBook;
    }

    private String concatUrlWithTitle(String url, String title) {
        return String.format(url, title);
    }
}

BestsellersService

package bookstore.scraper.book.scrapingtypeservice;

import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.book.Book;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookService;
import bookstore.scraper.fetcher.merlin.MerlinFetchingBookService;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.utilities.JSoupConnector;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.EnumMap;
import java.util.List;
import java.util.Map;


@Service
public class BestSellersService {

    private final EmpikUrlProperties empikUrlProperties;
    private final MerlinUrlProperties merlinUrlProperties;
    private final EmpikFetchingBookService empikBookService;
    private final MerlinFetchingBookService merlinBookService;
    private final JSoupConnector jSoupConnector;

    @Autowired
    public BestSellersService(EmpikFetchingBookService empikBookService, MerlinFetchingBookService merlinBookService, EmpikUrlProperties empikUrlProperties, MerlinUrlProperties merlinUrlProperties, JSoupConnector jSoupConnector) {
        this.empikBookService = empikBookService;
        this.merlinBookService = merlinBookService;
        this.empikUrlProperties = empikUrlProperties;
        this.merlinUrlProperties = merlinUrlProperties;
        this.jSoupConnector = jSoupConnector;
    }

    public Map<Bookstore, List<Book>> getBestSellers() {
        Map<Bookstore, List<Book>> bookstoreWithBestSellers = new EnumMap<>(Bookstore.class);

        bookstoreWithBestSellers.put(Bookstore.EMPIK, empikBookService
                .get5BestSellersEmpik(jSoupConnector.connect(empikUrlProperties.getEmpik().getBestSellers())));
        bookstoreWithBestSellers.put(Bookstore.MERLIN, merlinBookService
                .get5BestSellersMerlin(jSoupConnector.connect(merlinUrlProperties.getMerlin().getBestSellers())));

        return bookstoreWithBestSellers;
    }
}

TESTS

CategorizedBooksRankingServiceTest

package bookstore.scraper.rankingsystem;

import bookstore.scraper.book.Book;
import bookstore.scraper.book.scrapingtypeservice.CategorizedBookService;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.enums.CategoryType;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;

import java.util.List;
import java.util.Map;

import static bookstore.scraper.dataprovider.MerlinBookProvider.prepareExpectedRankingMap;
import static bookstore.scraper.dataprovider.MerlinBookProvider.prepareMapWithBookstoreAndCrimeBooks;
import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.when;

@RunWith(MockitoJUnitRunner.class)
public class CategorizedBooksRankingServiceTest {

    @Mock
    CategorizedBookService categorizedBookService;

    @InjectMocks
    CategorizedBooksRankingService categorizedBooksRankingService;

    @Test
    public void getRankingForCrimeCategory() {
        Map<Bookstore, List<Book>> bookstoreWith15CrimeBooks = prepareMapWithBookstoreAndCrimeBooks();

        when(categorizedBookService.get15BooksFromCrimeCategory()).thenReturn(bookstoreWith15CrimeBooks);

        Map<String, Integer> actualMap = categorizedBooksRankingService.getRankingForCategory(CategoryType.CRIME);
        Map<String, Integer> expectedMap = prepareExpectedRankingMap();

        assertEquals(expectedMap, actualMap);
    }
}

MerlinFetchingBookService

package bookstore.scraper.fetcher.merlin;

import bookstore.scraper.book.Book;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookServiceTest;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

import java.io.File;
import java.io.IOException;
import java.net.URISyntaxException;
import java.util.List;

import static bookstore.scraper.dataprovider.MerlinBookProvider.*;
import static org.junit.Assert.assertEquals;

@SpringBootTest
@RunWith(SpringRunner.class)
public class MerlinFetchingBookServiceTest {

    @Autowired
    private MerlinFetchingBookService merlinFetchingBookService;

    @Test
    public void getMostPreciseMerlinBook() throws IOException {
        File in = getFile("/merlin/MostPreciseBookMerlin.html");
        Document doc = Jsoup.parse(in, "UTF-8");

        Book actualBooks = merlinFetchingBookService.getMostPreciseMerlinBook(doc);
        Book expectedBooks = prepareMostPreciseBook();

        assertEquals(expectedBooks, actualBooks);
    }

    @Test
    public void get5BestSellersMerlin() throws IOException {
        File in = getFile("/merlin/BestsellersMerlin.html");
        Document doc = Jsoup.parse(in, "UTF-8");

        List<Book> actualBooks = merlinFetchingBookService.get5BestSellersMerlin(doc);
        List<Book> expectedBooks = prepare5Bestsellers();

        assertEquals(expectedBooks, actualBooks);
    }

    @Test
    public void get15BooksFromCategory() throws IOException {
        File in = getFile("/merlin/CrimeCategoryMerlin.html");
        Document doc = Jsoup.parse(in, "UTF-8");

        List<Book> actualBooks = merlinFetchingBookService.get15BooksFromCategory(doc);
        List<Book> expectedBooks = prepare15CrimeBooks();

        assertEquals(expectedBooks, actualBooks);
    }

    private File getFile(String resourceName) {
        try {
            return new File(EmpikFetchingBookServiceTest.class.getResource(resourceName).toURI());
        } catch (URISyntaxException e) {
            throw new IllegalStateException(e);
        }
    }
}

BestsellersServiceTest

package bookstore.scraper.book.scrapingtypeservice;

import bookstore.scraper.book.Book;
import bookstore.scraper.dataprovider.EmpikBookProvider;
import bookstore.scraper.dataprovider.MerlinBookProvider;
import bookstore.scraper.enums.Bookstore;
import bookstore.scraper.fetcher.empik.EmpikFetchingBookService;
import bookstore.scraper.fetcher.merlin.MerlinFetchingBookService;
import bookstore.scraper.urlproperties.EmpikUrlProperties;
import bookstore.scraper.urlproperties.MerlinUrlProperties;
import bookstore.scraper.utilities.JSoupConnector;
import org.jsoup.nodes.Document;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;

import java.util.List;
import java.util.Map;

import static bookstore.scraper.dataprovider.MergedBestsellersMapProvider.prepareExpectedMergedBestSellerMap;
import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;

@RunWith(MockitoJUnitRunner.class)
public class BestsellersServiceTest {

    @Mock
    private EmpikFetchingBookService empikBookService;
    @Mock
    private MerlinFetchingBookService merlinBookService;
    @Mock
    private EmpikUrlProperties empikUrlProperties;
    @Mock
    private MerlinUrlProperties merlinUrlProperties;
    @Mock
    private EmpikUrlProperties.Empik empikMock;
    @Mock
    private MerlinUrlProperties.Merlin merlinMock;
    @Mock
    JSoupConnector jSoupConnector;

    @InjectMocks
    private BestSellersService bestSellersService;

    @Test
    public void getBestSellers() {
        List<Book> merlinBestsellers = MerlinBookProvider.prepare5Bestsellers();
        List<Book> empikBestsellers = EmpikBookProvider.prepare5Bestsellers();
        Document empikDocument = mock(Document.class);
        Document merlinDocument = mock(Document.class);

        when(jSoupConnector.connect("https://www.empik.com/bestsellery/ksiazki")).thenReturn(empikDocument);
        when(empikUrlProperties.getEmpik()).thenReturn(empikMock);
        when(empikMock.getBestSellers()).thenReturn("https://www.empik.com/bestsellery/ksiazki");
        when(empikBookService.get5BestSellersEmpik(empikDocument)).thenReturn(empikBestsellers);

        when(jSoupConnector.connect("https://merlin.pl/bestseller/?option_80=10349074")).thenReturn(merlinDocument);
        when(merlinMock.getBestSellers()).thenReturn("https://merlin.pl/bestseller/?option_80=10349074");
        when(merlinUrlProperties.getMerlin()).thenReturn(merlinMock);
        when(merlinBookService.get5BestSellersMerlin(merlinDocument)).thenReturn(merlinBestsellers);

        Map<Bookstore, List<Book>> actualMap = bestSellersService.getBestSellers();
        Map<Bookstore, List<Book>> expectedMap = prepareExpectedMergedBestSellerMap();

        assertEquals(expectedMap, actualMap);
    }
}

Those are main classes that I want to be reviewed. Thanks a lot for each suggestion/opinion!