@ManyToMany after save of one entity list is refreshed after save of second isn't

Hello everybody my problem is pretty strange so let me show you part of code:

Part of AllegroFieldImpl:

@ManyToMany(targetEntity = AllegroCategoryImpl.class)
@JoinTable(name = "ALLEGRO_CATEGORIES_XREF", joinColumns = { 
        @JoinColumn(name = "ALLEGRO_FIELD_ID", nullable = false, updatable = false) }, 
        inverseJoinColumns = { @JoinColumn(name = "ALLEGRO_CATEGORY_ID", 
                nullable = false, updatable = false) })
@JsonBackReference
private List<AllegroCategory> allegroCategory = new ArrayList<AllegroCategory>();

Part of AllegroCategoryImpl:

@ManyToMany(fetch = FetchType.LAZY, cascade = CascadeType.ALL,targetEntity = AllegroFieldImpl.class)
@JoinTable(name = "ALLEGRO_CATEGORIES_XREF", joinColumns = { 
        @JoinColumn(name = "ALLEGRO_CATEGORY_ID", nullable = false, updatable = false) }, 
        inverseJoinColumns = { @JoinColumn(name = "ALLEGRO_FIELD_ID", 
                nullable = false, updatable = false) })
private List<AllegroField> allegroFields = new ArrayList<AllegroField>();

As you can see I have here 2 entities with ManyToMany relations model. Everything is working pretty good the lists are fetched etc. but there is one problem that I can't pass. When I create new AllegroFieldImpl and add for it existing AllegroCategoryImpl everything is saving on database well but when I execute method in AllegroCategoryImpl getAllegroFields() its looks like there is no update (I must restart server - maybe something is stored in catche ?). Here is code sample that will explain you all:

AllegroCategory ac = allegroService.readAllegroCategoryByID(1);
logger.error("Fields size: "+ac.getAllegroFields().size()); // 5

// Creating new AllegroField

AllegroField afnew = new AllegroFieldImpl();
afnew.setAllegroFieldId(421312);
afnew.setName("asdasdasdsa12312312312");
afnew.setObligatory(false);

afnew.getAllegroCategory().add(ac);
em.merge(afnew);

// Checking fields size
logger.error("Fields size: "+ac.getAllegroFields().size()); // 5

Any ideas how to solve this problem ?