Someone, somewhere, will have an opinion on why their chosen language is harder to learn than another. Often, these debates are kicked off by people who think they're experts but are just seeking to rationalise their own learning experiences.
What language is easy to learn for a German or Swedish speaker will not be the same for someone whose mother tongue is Japanese or Finnish. It really depends on several factors, such as the familiarity of the language you are learning, or the ease at which you can assimilate languages from that group.
If we just take standard English, yes, it is pretty simple to use and people forgive certain mistakes. But then when we see some of the huge differences in meaning after minute changes to the wording, we realise English is actually not as straightforward as all that. Bear with me as I try to demonstrate what I mean...
1. verb + to- vs verb + _ing
What's the difference between "remember feeding the cat" and "remember to feed the cat"? Or "stop eating" and "stop to eat"? Think about it for a second.
"I remember feeding the cat" suggests you did it before and now you are recalling the moment, whereas "I remembered to feed the cat" means you didn't forget to do it.
"I stopped eating at 1" implies that you finished your meal then.
"I stopped to eat at 1" says that you dropped what you were doing in order to eat at 1.
With the verb "like", it gets even more freaky.
"I like cycling" and "I like to cycle" on their own mean roughly the same thing. But if I add some elements, we see how wildly different they can become:
"I like cycling to the shops" - here, we say that we enjoy it.
"I like to cycle to the shops" - in this one, we imply that it's not necessarily a pleasure for us, but we do it because it's practical or the better option.
It becomes more noticeable in something like household chores:
"I like to vacuum the house every weekend" vs "I like vacuuming the house every weekend" shows a very large gap in meaning. In the first, it's a job we do because otherwise the place would be very messy. In the second, we are demonstrating our utter madness in actually declaring our pleasure in the task.
2. hard vs hardly
"You work hard" or "you hardly work" are worlds apart in meaning. The former means you don't rest or take many holidays, and the latter means you are either a banker or a civil servant.
Joke.
Well no, it was a dig.
3. opposites
"The alarm went off at 3 in the morning" is completely different to "the lights went off at 3 in the morning." In fact, they are total opposites - we call these contranyms, as they are words or phrases that have opposite meanings. Others include "sanction", "refrain", "dust" or "cleave".
For the clarity, "sanction" can mean either to take away permission or to give permission; "refrain" means either to avoid doing something, or to repeat; "dust" means to add fine particles or to remove them; and "cleave" means to stick to things together, or to separate them.
4. homonyms everywhere
Let us not forget these beauties... homonyms run throughout the English language, and are only really distinguishable by the context in which they are found. They can be divided into homophones (words that sound the same but are spelled differently or have two meanings) or homographs (words that are spelled the same but mean different things, and may be pronounced differently too).
Homographs - think about these words and how they can change meaning depending on the context (possible answers below):
right
bark
row
current
record
wound
rock
stick
like
figure
tablet
Homophones - say these words to another person and ask them how to spell them:
ate / eight
fare / fair
their / they're / there
meat / meet
pray / prey
knows / noes / nose
week / weak
tow / toe
to / too / two
made / maid
5. the and other articles
You might not believe this, but such a small word can create an almighty headache for a lot of learners. For example:
farmers produce crops / the farmers produce crops / the farmers produce the crops / some farmers produce crops
cars were invented in the 19th century / the car was invented in the 19th century / the car is in the garage / some cars are in the garage
In some languages, articles are used everywhere, such as French or Italian, and they can come in various forms, such as masculine/feminine and singular/plural. In others, like Czech or Slovenian, they exist but they have lost their significance, so nobody bothers using them. Then there's English, where they are used sometimes but not everywhere, and change meaning depending on how they are used.
Consider the following:
We should look after the old and infirm
We are building a shelter for the homeless
This is a place to heal the sick
This is a worrying development for the unemployed
These expressions refer to specific groups of people and their issues. They are used almost like nouns, or at least they allude to nouns, even if they are not. Yet what is most bizarre about this is that we can't say "the fit", "the intelligent", "the happy" or "the independent". Seems a bit discriminatory to me...
But then, what is the difference between:
breakfast / the breakfast
lunch / some lunch
dinner / a dinner
Without the article, we are referring merely to the daily meal:
"We have breakfast / lunch / dinner every day."
But by adding an article, we change the context entirely:
"The breakfast in that hotel is very filling."
"We had some lunch before we carried on working."
"I need to organise a dinner for 40 people next Friday."
We say "the Pacific Ocean", "the Sahara Desert", "the Black Forest", "the River Thames", but "Lake Titicaca", "Mount Olympus", and "Blueberry Hill".
6. Future
When someone says "I'm going to get some food shortly", rather than "I will get some food shortly", what do they mean? It's all in the context, of course.
"I'm going to get some food shortly, so let me know what you want" - this implies you have planned this, and your mission to the shop is imminent.
"I see everyone looks a bit tired, so I'll get some food shortly." - in this case, you hadn't planned it, but you're taking the decision now to get some.
And this is just one of many examples of where the future in English relies on what you wish to imply. English is an implied language, meaning that we can often deduce what someone says or writes simply by the words used. Of course, intonation, gesticulations and facial expression help in speech, but that doesn't stop it from being a real minefield.
7. Intonation
In English, people tend to have conversations that heavily rely on the melody for feeling. Let me explain: in yes/no questions, it is common that the last word or syllable will have a rising tone.
"Are you Swedish?" 
"Do you like fish?"
"Did you have a lovely time?"
But if you ask an open question, especially a personal one, the last word or syllable will often have a falling tone.
"Where do you live?" 
"What's your name?"
"How old are you?"
And the reason is actually quite a practical one...
By using a rising intonation in yes/no questions, we are asking for confirmation, so just a straightforward yes or no will suffice.
When we use a falling intonation in open questions, we simply want to ask for facts.
But then, if we swap them around, mad things happen...
By using a falling intonation in yes/no questions, we can seem very rude. We are either demonstrating that we aren't really interested in the answer, or we know the answer but are expected to ask it anyhow.
By using a rising intonation in open questions, we can come across as a little too eager to know the answer. This can either sound like a sleazy move by the questioner, or an intrusive interrogation of the other.
So there you go. These are just a few of the hurdles learners have to clear on their journeys.
It's a strange phenomenon, but learning languages like German and Polish can seem intimidating to learners starting out, but they get easier the more you know. English is the opposite. It all looks pretty straightforward, and then when you get to about B2 level, you start learning finicky little differences where you are confronted with the reality of the daunting task in hand. There is one upside though: it's easy to speak bad English, but most people won't give you a rough time over it!
ANSWERS:
right = your right to remain silent / turn right at the lights / you're right about everything
bark = the dogs bark all night / we put bark on the garden to stop the weeds from growing
row = three trees in a row / I had a row with my neighbours / I got in a boat and learned how to row
current = the current problems are huge / the current took the boat out to sea
record = we must record all the details / she's broken the world record / I have a huge record collection
wound = I wound the rope round the tree / the dog picked up a wound somehow
rock = all boats rock on the sea / I threw a rock into the river / rock music is still very popular
stick = stick the paper to the wall / I threw a stick to the dog / please stick around / stick it in the bin
like = you look like your mother / I like chocolate / we have a policy of like for like
figure = he earned a six-figure sum / there was a strange figure in the forest / figure out this problem
tablet = I took a tablet for my headache / they found an ancient tablet / my tablet has many apps
